UNITED STATES SECURITIES AND EXCHANGE COMMISSION

Washington, D.C. 20549

Form 10-K

Commission File Number 1-31340

The Cato Corporation

Registrant

Securities registered pursuant to Section 12(b) of the Act:

Class A Common Stock

Preferred Share Purchase Rights

Securities registered pursuant to Section 12(g) of the Act:

None

     Indicate by check mark whether the Registrant (1) has filed all reports required to be filed by Section 13 or 15(d) of The Securities Exchange Act of 1934 during the preceding 12 months (or for such shorter period that the Registrant was required to file such reports), and (2) has been subject to such filing requirements for the past 90 days.     Yes þ          No o

     Indicate by check mark, if disclosure of delinquent filers pursuant to Item 405 of the Regulation S-K is not contained herein, and will not be contained, to the best of the Registrant’s knowledge, in definitive proxy or information statements incorporated by reference in Part III of this Form 10-K or any amendment to this Form 10-K. o

     Indicate by check mark whether the registrant is an accelerated filer (as defined in Rule 12b-2 of the Act).     Yes þ          No o

     The aggregate market value of the Registrant’s Class A Common Stock held by Non-affiliates of the Registrant as of July 31, 2004, the last business day of the Company’s most recent second quarter, was $420,193,294 based on the last reported sale price per share on the New York Stock Exchange (NYSE) on that date.

     As of March 29, 2005, there were 20,367,720 shares of Class A Common Stock and 460,350 shares of Convertible Class B Common Stock outstanding.

DOCUMENTS INCORPORATED BY REFERENCE

     Portions of the proxy statement relating to the 2005 annual meeting of shareholders are incorporated by reference into the following part of this annual report:

Part III — Items 10, 11, 12, 13 and 14

(This page intentionally left blank)

THE CATO CORPORATION

FORM 10-K

TABLE OF CONTENTS

Forward-looking Information

      The following discussion and analysis should be read along with the Consolidated Financial Statements, including the accompanying Notes appearing later in this report. Any of the following are “forward-looking” statements within the meaning of Section 27A of the Securities Act of 1933, as amended, and Section 21E of the Securities Exchange Act of 1934, as amended: (1) statements in this Annual Report on Form 10-K that reflect projections or expectations of our future financial or economic performance; (2) statements that are not historical information; (3) statements of our beliefs, intentions, plans and objectives for future operations, including those contained in “Business”, “Properties”, “Legal Proceedings”, “Controls and Procedures” and “Management’s Discussion and Analysis of Financial Condition and Results of Operations”; (4) statements relating to our operations or activities for 2005 and beyond; and (5) statements relating to our future contingencies. Words such as “expects”, “anticipates”, “approximates”, “believes”, “estimates”, “hopes”, “intends”, “may”, “plans”, “should” and variations of such words and similar expressions are intended to identify such forward-looking statements. No assurance can be given that actual results or events will not differ materially from those projected, estimated, assumed or anticipated in any such forward-looking statements. Forward-looking statements included in this report are based on information available to us as of the filing date of this report, and we assume no obligation to update any such forward-looking information contained in this report.

      Our website is located at www.catocorp.com. We make available free of charge, through our website, our annual reports on Form 10-K, quarterly reports on Form 10-Q, current reports on Form 8-K, proxy statements and other reports (including amendments to these reports) filed or furnished pursuant to Section 13(a) or 15(d) under the Securities Exchange Act of 1934. These reports are available as soon as reasonably practicable after we electronically file those materials with the SEC. We also post on our website the charters of our Audit, Compensation and Corporate Governance and Nominating Committees; our Corporate Governance Guidelines, Code of Business Conduct and Ethics; and any amendments or waivers thereto; and any other corporate governance materials contemplated by SEC or New York Stock Exchange regulations. The documents are also available in print to any shareholder who requests by contacting our corporate secretary at our company offices.

PART I

General

      The Company, founded in 1946, operated 1,177 women’s fashion specialty stores at January 29, 2005, under the names “Cato,” “Cato Fashions,” “Cato Plus” and “It’s Fashion!” in 29 states, principally in the southeastern United States. The Company offers quality fashion apparel and accessories at low prices, every day in junior/ missy and plus sizes. Additionally, the Company offers clothing for girls sizes 7 to 16 in selected locations. The Company’s stores feature a broad assortment of apparel and accessories, including casual and dressy sportswear, dresses, careerwear, coats, shoes, costume jewelry and handbags. A major portion of the Company’s merchandise is sold under its private labels and is produced by various vendors in accordance with the Company’s specifications. Most stores range in size from 4,000 to 6,000 square feet and are located primarily in strip shopping centers anchored by national discounters or market-dominant grocery stores. The Company emphasizes friendly customer service and coordinated merchandise presentations in an appealing store environment. The Company offers its own credit card and layaway plan. Credit and layaway sales represented 14% of retail sales in fiscal 2004. See Note 14 to the Consolidated Financial Statements, “Reportable Segment Information” for a discussion of segment information.

Restatement of Prior Financial Information

      We have restated the consolidated balance sheet at January 31, 2004, and the consolidated statements of income, cash flows and stockholders’ equity for the years ended January 31, 2004 and February 1, 2003 in this Annual Report on Form 10-K. We have also restated the quarterly financial information for fiscal 2003 and the first three quarters of fiscal 2004. See Note 13 to the accompanying consolidated financial statements. The restatement also affects periods prior to fiscal 2002. The impact of the restatement on such prior periods has been reflected as an adjustment to retained earnings as of February 2, 2002 in the accompanying consolidated statements of stockholders’ equity. We have also restated the applicable financial information for fiscal 2000, fiscal 2001, fiscal 2002 and fiscal 2003 in “Item 6. Selected Financial Data.” The restatement corrects our historical lease accounting practices. For information with respect to the restatement, see Note 1 to the accompanying consolidated financial statements. We did not amend our previously filed Annual Report on Form 10-K or Quarterly Reports on Form 10-Q for the restatement, and the financial statements and related financial information contained in such reports should no longer be relied upon. Throughout this Form 10-K all referenced amounts for prior periods and prior period comparisons reflect the balances and amounts on a restated basis.

Business

      The Company’s primary objective is to be the leading fashion specialty retailer for fashion and value conscious females in its markets. Management believes the Company’s success is dependent upon its ability to differentiate its stores from department stores, mass merchandise discount stores and competing women’s specialty stores. The key elements of the Company’s business strategy are:

      Merchandise Assortment. The Company’s stores offer a wide assortment of apparel and accessory items in junior/ missy and plus sizes and emphasize color, product coordination and selection.

      Value Pricing. The Company offers quality merchandise that is generally priced below comparable merchandise offered by department stores and mall specialty apparel chains, but is generally more fashionable than merchandise offered by discount stores. Management believes that the Company has positioned itself as the everyday low price leader in its market segment.

      Strip Shopping Center Locations. The Company locates its stores principally in convenient strip centers anchored by national discounters or market-dominant grocery stores that attract large numbers of potential customers.

      Customer Service. Store managers and sales associates are trained to provide prompt and courteous service and to assist customers in merchandise selection and wardrobe coordination.

      Credit and Layaway Programs. The Company offers its own credit card and a layaway plan to make the purchase of its merchandise more convenient.

      Expansion. The Company plans to continue to expand into northern, midwestern and western adjacent states, as well as continuing to fill-in existing southeastern core geography.

Merchandising

      The Company offers a broad selection of high quality and exceptional value apparel and accessories to suit the various lifestyles of the fashion and value conscious females. In addition, the Company offers on-trend fashion in exciting colors with consistent fit and quality.

      The Company’s merchandise lines include dressy, career, and casual sportswear, dresses, coats, shoes, lingerie, costume jewelry and handbags. Apparel for girls sizes 7 to 16 is offered in approximately 1,000 stores. The Company primarily offers exclusive merchandise with fashion and quality comparable to mall specialty stores at low prices, every day.

      The collaboration of the merchandising team with an expanded in-house product development and direct sourcing function has enhanced merchandise offerings delivering quality exclusive products at lower costs. The product development and direct sourcing operations provide research on emerging fashion and color trends, technical services and direct sourcing options.

      As a part of its merchandising strategy, members of the Company’s merchandising staff frequently visit selected stores, monitor the merchandise offerings of other retailers, regularly communicate with store operations associates and frequently confer with key vendors. The Company tests most new fashion-sensitive items in selected stores to aid it in determining their appeal before making a substantial purchasing commitment. The Company also takes aggressive markdowns on slow-selling merchandise and does not carry over merchandise to the next season.

      Although the Company purchases merchandise from approximately 1,500 suppliers, most of its merchandise is purchased from approximately 100 primary vendors. In fiscal 2004, purchases from the Company’s largest vendor accounted for approximately 6% of the Company’s total purchases. No other vendor accounted for more than 3% of total purchases. The Company is not dependent on its largest vendor or any other vendor for merchandise purchases and the loss of any single vendor or group of vendors would not have a material adverse effect on the Company’s operating results or financial condition. A substantial portion of the Company’s merchandise is sold under its private labels and is produced by various vendors in accordance with the Company’s strict specifications. The Company purchases most of its merchandise from domestic importers and vendors, which typically minimizes the time necessary to purchase and obtain shipments in order to enable the Company to react to merchandise trends in a more timely fashion. Although a significant portion of the Company’s merchandise is manufactured overseas, principally in the Far East, any economic, political or social unrest in any one region is not expected to have a material adverse effect on the Company’s ability to obtain adequate supplies of merchandise.

      An important component of the Company’s strategy is the allocation of merchandise to individual stores based on an analysis of sales trends by merchandise category, customer profiles and climatic conditions. A merchandise control system provides current information on the sales activity of each merchandise style in each of the Company’s stores. Point-of-sale terminals in the stores collect and transmit sales and inventory information to the Company’s central database, permitting timely response to sales trends on a store-by-store basis.

      All merchandise is shipped directly to the Company’s distribution center in Charlotte, North Carolina, where it is inspected and then allocated by the merchandise distribution staff for shipment to individual stores. The flow of merchandise from receipt at the distribution center to shipment to stores is controlled by an on-line system. Shipments are made by common carrier, and each store receives at least one shipment per week.

      The Company uses radio, in store signage, graphics and a Company website as its primary advertising media. The Company uses radio advertising in selected trade areas. The Company’s total advertising expenditures were approximately .8% of retail sales in fiscal 2004.

Store Operations

      The Company’s store operations management team consists of 1 director of stores, 4 territorial managers, 15 regional managers and 120 district managers. Regional managers receive a salary plus a bonus based on achieving targeted goals for sales, payroll, shrinkage control and store profitability. District managers receive a salary plus a bonus based on achieving targeted objectives for district sales increases and shrinkage control. Stores are staffed with a manager, two assistant managers and additional part-time sales associates depending on the size of the store and seasonal personnel needs. Store managers receive a salary and all other store personnel are paid on an hourly basis. Store managers, assistant managers and sales associates are eligible for monthly and semi-annual bonuses based on achieving targeted goals for their store’s sales increases and shrinkage control.

      The Company is constantly improving its training programs to develop associates. Over 80% of store and field management are promoted from within, allowing the Company to internally staff an expanding store base. The Company has training programs at each level of store operations. New store managers are trained in training stores managed by experienced associates who have achieved superior results in meeting the Company’s goals for store sales, payroll expense and shrinkage control. The type and extent of district manager training varies depending on whether the district manager is promoted from within or recruited from outside the Company. All district managers receive at a minimum a one-week orientation program at the Company’s corporate office.

Store Locations

      Most of the Company’s stores are located in the southeastern United States in a variety of markets ranging from small towns to large metropolitan areas with trade area populations of 20,000 or more. Stores range in size from 4,000 to 6,000 square feet and average approximately 4,500 square feet.

      All of the Company’s stores are leased. Approximately 93% are located in strip shopping centers and 7% in enclosed shopping malls. The Company locates stores in strip shopping centers anchored by a national discounter, primarily Wal-Mart Supercenters, or market-dominant grocery stores. The Company’s strip center locations provide ample parking and shopping convenience for its customers.

      The Company’s store development activities consist of opening new stores in new and existing markets, and relocating selected existing stores to more desirable locations in the same market area. The following table sets forth information with respect to the Company’s development activities since fiscal 2000.

Store Development

      In Fiscal 2004 the Company relocated 29 stores and remodeled 17 stores.

      In Fiscal 2005 the Company plans to open approximately 90 new stores, relocate 20 stores, close 10 stores, and remodel 15 stores.

      The Company periodically reviews its store base to determine whether any particular store should be closed based on its sales trends and profitability. The Company intends to continue this review process to close underperforming stores. The five stores closed in 2004 were not material to the Company’s results of operations.

Credit and Layaway

      The Company offers its own credit card, which accounted for approximately 9% of retail sales in fiscal 2004. The Company’s net bad debt expense in fiscal 2004 was 7.3% of credit sales.

      Customers applying for the Company’s credit card are approved for credit if they have a satisfactory credit record and meet minimum income criteria. Customers are required to make minimum monthly payments based on their account balances. If the balance is not paid in full each month, the Company assesses the customer a finance charge. If payments are not received on time, the customer is assessed a late fee.

      Under the Company’s layaway plan, merchandise is set aside for customers who agree to make periodic payments. The Company adds a nonrefundable administrative fee to each layaway sale. If no payment is made for four weeks, the customer is considered to have defaulted, and the merchandise is returned to the selling floor and again offered for sale, often at a reduced price. All payments made by customers who subsequently default on their layaway purchase are returned to the customer upon request, less the administrative fee and a restocking fee. The Company defers recognition of layaway sales and its related fees to the accounting period when the customer picks up layaway merchandise. Layaway sales represented approximately 5% of retail sales in fiscal 2004, 2003 and 2002.

Management Information Systems

      The Company’s systems provide daily financial and merchandising information that is used by management to enhance the timeliness and effectiveness of purchasing and pricing decisions. Management uses a daily report comparing actual sales with planned sales and a weekly ranking report to monitor and control purchasing decisions. Weekly reports are also produced which reflect sales, weeks of supply of inventory and other critical data by product categories, by store and by various levels of responsibility reporting. Purchases are made based on projected sales but can be modified to accommodate unexpected increases or decreases in demand for a particular item.

      Sales information is projected by merchandise category and, in some cases, is further projected and actual performance measured by stock keeping unit (SKU). Merchandise allocation models are used to distribute merchandise to individual stores based upon historical sales trends, climatic differences, customer demographic differences and targeted inventory turnover rates.

Competition

      The women’s retail apparel industry is highly competitive. The Company believes that the principal competitive factors in its industry include merchandise assortment and presentation, fashion, price, store location and customer service. The Company competes with retail chains that operate similar women’s apparel specialty stores. In addition, the Company competes with mass merchandise chains, discount store chains and major department stores. To the extent that the Company opens stores in larger cities and metropolitan areas, competition is expected to be more intense in those markets.

Regulation

      A variety of laws affect the revolving credit program offered by the Company. The Federal Consumer Credit Protection Act (Truth-in Lending) and Regulation Z promulgated thereunder require written disclosure of information relating to such financing, including the amount of the annual percentage rate and the finance charge. The Federal Fair Credit Reporting Act also requires certain disclosures to potential customers concerning credit information used as a basis to deny credit. The Federal Equal Credit Opportunity Act and Regulation B promulgated thereunder prohibit discrimination against any credit applicant based on certain specified grounds. The Federal Trade Commission has adopted or proposed various trade regulation rules dealing with unfair credit and collection practices and the preservation of consumers’ claims and defenses. The Company is also subject to the U.S. Patriot Act and the Bank Secrecy Act which require the Company to monitor account holders and account transactions, respectively. Additionally, the Gramm-Leach-Bliley Act requires the Company to disclose, initially and annually, to its customers, the Company’s privacy policy as it relates to a customer’s non-public personal information.

Associates

      As of January 29, 2005, the Company employed approximately 9,600 full-time and part-time associates. The Company also employs additional part-time associates during the peak retailing seasons. The Company is not a party to any collective bargaining agreements and considers that its associate relations are good.

      The Company’s distribution center and general offices are located in a Company-owned building of approximately 492,000 square feet located on a 15-acre tract in Charlotte, North Carolina. The Company’s automated merchandise handling and distribution activities occupy approximately 418,000 square feet of this building and its general offices and corporate training center are located in the remaining 74,000 square feet. A building of approximately 24,000 square feet located on a 2-acre tract adjacent to the Company’s existing location is used for receiving and staging shipments prior to processing.

      Substantially all of the Company’s retail stores are leased from unaffiliated parties. Most of the leases have an initial term of five years, with two to three five-year renewal options. Many of the leases provide for fixed rentals plus a percentage of sales in excess of a specified volume.

      From time to time, claims are asserted against the Company arising out of operations in the ordinary course of business. The Company currently is not a party to any pending litigation that it believes is likely to have a material adverse effect on the Company’s financial conditions or results of operations.

      None.

      The executive officers of the Company and their ages as of March 31, 2005 are as follows:

      John P. Derham Cato has been employed as an officer of the Company since 1981 and has been a director of the Company since 1986. Since January 2004, he has served as Chairman, President and Chief Executive Officer. From May 1999 to January 2004, he served as President, Vice Chairman of the Board and Chief Executive Officer. From June 1997 to May 1999, he served as President, Vice Chairman of the Board and Chief Operating Officer. From August 1996 to June 1997, he served as Vice Chairman of the Board and Chief Operating Officer. From 1989 to 1996, he managed the Company’s off-price division, serving as Executive Vice President and as President and General Manager of the It’s Fashion! Division from 1993 to August 1996. Mr. John Cato is currently a director of Ruddick Corporation.

      Michael O. Moore has been employed by the Company as Executive Vice President, Chief Financial Officer and Secretary since July 1998 and has been a director of the Company since 2002. Mr. Moore served as Vice President, Chief Financial Officer for Party Experience from 1997 to 1998, Executive Vice President, Chief Financial Officer of David’s Bridal from 1994 to 1997, and was employed by Bloomingdales from 1984 to 1994 serving as Senior Vice President, Chief Financial Officer from 1990 to 1994.

      B. Allen Weinstein joined the Company as Executive Vice President, Chief Merchandising Officer of the Cato Division in August 1997. Since November 2004, he has served as Executive Vice President, Chief Merchandising Officer of the Company. From 1995 to 1997, he was Senior Vice President — Merchandising of Catherines Stores Corporation. From 1981 to 1995, he served as Senior Vice President of Merchandising for Beall’s, Inc.

      Howard A. Severson has been employed by the Company since 1985. Since January 1993, he has served as Executive Vice President, Chief Real Estate and Store Development Officer and Assistant Secretary. From 1993 to 2001 Mr. Severson also served as a director. From August 1989 through January 1993, Mr. Severson served as Senior Vice President — Chief Real Estate Officer.

      Michael T. Greer has been employed by the Company since 1985. Since November 2004, he has served as Senior Vice President, Director of Stores of the Company. From February 2004 through November 2004, he served as Senior Vice President, Director of Stores of the Cato Division. From 2002 to 2003 Mr. Greer served as Vice President, Director of Stores of the It’s Fashion! Division. From 1999 to 2001 he served as Territorial Vice President of Stores of the Cato Division and from 1996 to 1999 he served as Regional Vice President of Stores of the Cato Division. From 1985 to 1995, Mr. Greer held various store operational positions in the Cato Division.

      Robert C. Brummer joined the Company as Senior Vice President, Human Resources and Assistant Secretary in January 2001. From 1999 through 2000, he was employed by Sleepy’s, a beddings specialty retailer, as Vice President, Human Resources and Payroll. From 1997 through 1998, he was Vice President, Human Resources and Loss Prevention for The Party Experience, a party supplies specialty retailer. From 1995 until 1997, he was Vice President, Human Resources and Loss Prevention for No Body Beats The Wiz, an electronics specialty store chain.

PART II

Market & Dividend Information

      The Company’s Class A Common Stock trades on the New York Stock Exchange (NYSE) under the symbol CTR. As required by Section 3.03A.12(a) of the NYSE listing standards, The Cato Corporation filed with the NYSE the certification of its Chief Executive Officer that he is not aware of any violation by the company of NYSE corporate governance listing standards. Below is the market range and dividend information for the four quarters of fiscal 2004 and 2003.

      As of March 29, 2005 the approximate number of record holders of the Company’s Class A Common Stock was 1,279 and there were 3 record holders of the Company’s Class B Common Stock.

      Certain selected financial data for the five fiscal years ended January 29, 2005 have been derived from the Company’s audited financial statements. The financial statements and Independent Registered Public Accounting Firm’s reports for the three most recent fiscal years are contained elsewhere in this report. All data set forth below are qualified by reference to, and should be read in conjunction with, the Company’s Consolidated Financial Statements (including the Notes thereto) and “Management’s Discussion and Analysis of Financial Condition and Results of Operations” appearing elsewhere in this annual report.

      The five-year selected consolidated financial data presented in this Item 6 has been revised to reflect a restatement. For information with respect to the restatement, see Note 1 to the accompanying consolidated financial statements.

Restatement of Prior Financial Information

      We have restated the consolidated balance sheet at January 31, 2004, and the consolidated statements of income, cash flows and stockholders’ equity for the years ended January 31, 2004 and February 1, 2003 in this Annual Report on Form 10-K to correct our historical lease, inbound freight capitalization and vendor allowance accounting practices. We have also restated our quarterly financial information for fiscal 2003 and the first three quarters of fiscal 2004. See Note 13 to the accompanying consolidated financial statements. The restatement also affects periods prior to fiscal 2002. The impact of the restatement on such prior periods has been reflected as an adjustment of $7.3 million to retained earnings as of February 2, 2002 in the accompanying consolidated statement of stockholders’ equity. We have also restated the applicable financial information for fiscal 2000, fiscal 2001, fiscal 2002 and fiscal 2003 in “Item 6. Selected Financial Data.”

      After the staff of the Securities and Exchange Commission issued a letter on February 7, 2005 we, like many other retailers, reviewed our lease accounting practices and determined that certain corrections were needed. As a result, we corrected our lease accounting practices for fiscal 2004 and restated certain historical financial information. The restatement corrections did not impact cash payments and had no impact on revenues, comparable store sales or operating cash flows.

      The Company corrected its lease accounting practices to recognize lease expense on a straight-line basis over the expected lease term (as that term is defined by Statement of Financial Accounting No. 13, as amended “SFAS No. 13”) beginning on the date the Company takes possession of the leased property, including lease renewal periods that are required to be included in the lease term because of economic penalties that result in the renewal being reasonably assured. Likewise, the Company corrected its practices to recognize landlord allowances on a straight-line basis over the lease term.

      The restatement includes adjustments to cost of goods sold, gross margin, operating income, income before taxes, income tax provision, net income and earnings per share. This correction to our lease accounting practices reduced net income by $484,000 and diluted earnings per share by $0.02 in fiscal 2004. The corrections decreased net income by $775,000 or $0.03 per diluted share in fiscal year 2003 and by $366,000 or $0.02 per diluted share in fiscal 2002. In addition, the Company increased net income by $400,000 or $0.01 per diluted share in fiscal 2003 and decreased net income by $22,000 in fiscal 2002 to properly capitalize inbound freight on domestic purchases and to properly account for vendor allowances.

      For information with respect to the restatement adjustments, see Note 1 to the accompanying consolidated financial statements.

      We did not amend our previously filed Annual Reports on Form 10-K for fiscal years 2003 and 2002 or Quarterly Reports on Form 10-Q for fiscal year 2004 for the restatement, and, accordingly, the financial statements and related financial information contained in such reports should no longer be relied upon.

      Throughout “Management’s Discussion and Analysis of Financial Condition and Results of Operations,” all referenced amounts for prior periods and prior period comparisons reflect the balances and amounts on a restated basis.

Results of Operations

      The table below sets forth certain financial data of the Company expressed as a percentage of retail sales for the years indicated: