Exhibit 10.5

EXECUTION COPY

TAX INDEMNIFICATION AGREEMENT dated as of November 4, 2010 (this “Agreement”), between THE FRESH MARKET, INC. (the “Company”) and all of the shareholders identified on the signature pages of this Agreement.

WHEREAS the Company has elected to be an S-corporation (the “S Election”) under the Internal Revenue Code of 1986 or the Internal Revenue Code of 1954, as applicable, and in each case as amended (the “Code”);

WHEREAS The Fresh Market of Massachusetts, Inc. (the “Subsidiary”) is a wholly owned subsidiary of the Company and has elected to be a qualified subchapter S subsidiary (the “QSub Election”) under the Code;

WHEREAS the Company intends to conduct an initial public offering (the “IPO”) and, in connection with the IPO, the Company’s S Election and the Subsidiary’s QSub Election will terminate, and each of the Company and the Subsidiary will each be treated as C-corporations under the Code;

WHEREAS at all times the Company’s S Election was in effect, the Shareholders (defined below) paid Federal and certain state and local income taxes on their allocable share of the Company’s Taxable Income (defined below) as determined under the Code and certain equivalent state or local statutes, and the Shareholders will continue to pay such taxes (as they become due) for such periods as the Company’s S Election remains in effect;

WHEREAS the Company is obligated under its shareholder agreement to make pro rata distributions to its Shareholders in amounts equal to the Shareholders’ estimated tax liability, calculated as if each Shareholder would be taxable on its allocable share of the Company’s Taxable Income at the maximum Federal income tax rate and the maximum state and local income tax rates; and

WHEREAS the Company and the Shareholders desire to set forth their agreement that the Company shall bear the risk of any additional tax liability, as well as any related losses, costs and expenses, resulting from (i) any statement or restatement of the Company’s Taxable Income on any income tax return or (ii) any Determination (as defined below), in each case for any open taxable period beginning before the termination of the Company’s S Election;

NOW, THEREFORE, in consideration of the mutual covenants and agreements contained herein, the parties hereby agree as follows:

ARTICLE I

Definitions

SECTION 1.01. General. The following terms shall have the following meanings (such meanings to apply equally to the singular and plural forms of the terms

defined). All section references are to this Agreement unless otherwise stated. All references to “includes” and “including” mean “includes without limitation” or “including without limitation”, as the case may be.

SECTION 1.02. Definition of Terms.

“Company’s Taxable Income” means the Company’s taxable income, combined with the Subsidiary’s taxable income, as applicable.

“Determination” means the final resolution of liability for any tax for any taxable period as a result of (i) a “determination” as defined in Treasury Regulation § 1.1377-2(c), (ii) a final determination made by a competent Taxing Authority or (iii) the payment of tax by the Shareholders if the Shareholders and the Company agree that the payment should be made and no action should be taken to recoup that payment.

“Proceeding” means any proceeding that will potentially give rise to a Determination.

“Shareholder” means for any taxable period a person who was a shareholder of the Company during all or part of such taxable period.

“taxes” means all Federal, state and local taxes, assessments, duties or similar charges of any kind whatsoever, including any interest, additions to tax or penalties applicable thereto.

“Taxing Authority” means any governmental body charged with the determination, collection or imposition of taxes.

ARTICLE II

Payments and Indemnity

SECTION 2.01. Tax Returns. Upon filing any Federal, state or local income tax return (amended or otherwise) for any taxable period during which the Company had an S Election in effect, the Company shall calculate each Shareholder’s estimated tax liability for such taxable period as if each such Shareholder would be taxable on its allocable share of the Company’s Taxable Income at the maximum Federal income tax rate and the maximum state and local income tax rates applicable to each such Shareholder; provided, that in any case where another person or entity is directly taxed on a Shareholder’s income, such Shareholder’s estimated tax liability shall be determined by reference to such other person or entity. The Company shall calculate for each Shareholder the excess of such Shareholder’s estimated tax liability over the amount previously distributed by the Company to such Shareholder in respect of tax liabilities for the relevant taxable period (the “2.01 Excess”). The Company shall make a payment to each Shareholder in proportion to each Shareholder’s shareholdings during the relevant taxable period in an amount sufficient so that the Shareholder with the highest 2.01 Excess receives a payment equal thereto. Payments made pursuant to this Section 2.01 shall be made at the time the relevant income tax return is filed.

2

SECTION 2.02. Determinations. After any Determination, the Company shall calculate each Shareholder’s estimated tax liability for the relevant taxable period as if each such Shareholder would be taxable on its allocable share of the Company’s Taxable Income at the maximum Federal income tax rate and the maximum state and local income tax rates applicable to each such Shareholder; provided, that in any case where another person or entity is directly taxed on a Shareholder’s income, such Shareholder’s estimated tax liability shall be determined by reference to such other person or entity. The Company shall calculate for each Shareholder the excess of such Shareholder’s estimated tax liability over the amount previously distributed by the Company to such Shareholder in respect of taxes for such taxable period (the “2.02 Excess”). The Company shall make a payment to each Shareholder in proportion to each Shareholder’s shareholdings during the relevant taxable period in an amount sufficient so that the Shareholder with the highest 2.02 Excess receives a payment equal thereto. Payments made pursuant to this Section 2.02 shall be made within 120 days of the relevant Determination.

SECTION 2.03. Indemnification. The Company shall indemnify and hold harmless the Shareholders from any losses, costs or expenses (including reasonable attorneys’ fees) arising out of any claims made pursuant to Section 2.01 or Section 2.02. Payments made pursuant to this Section 2.03 shall be made at the same time as the payment made pursuant to Section 2.01 or Section 2.02, as applicable.

ARTICLE III

Notice, Proceedings and Inconsistent Reporting

SECTION 3.01. Notice and Proceedings. (a) Any Shareholder that believes it may be entitled to a payment under this Agreement as a result of a Proceeding shall use reasonable efforts to promptly notify the Company of such Proceeding.

(b) The Company will have the option to represent itself in any Proceeding, at its own expense and using advisors of the Company’s choice.

(c) Each Shareholder shall cooperate fully with the Company in any Proceeding and shall have the right, but not the obligation, to participate in such Proceeding at its own expense.

(d) Breach by any Shareholder of any of the provisions of this Section 3.01 will terminate the Company’s obligation to make payments to such Shareholder under Article II, to the extent any such breach materially prejudices the result of any Proceeding.

SECTION 3.02. Inconsistent Reporting. If a Shareholder hereafter reports an item on such Shareholder’s income tax return in a manner materially inconsistent with the tax treatment reflected in the Schedule K-1 or other tax information provided to the Shareholder by the Company for a taxable period during which the Company had an S Election in effect, such Shareholder shall notify the Company of such

3

treatment before filing such Shareholder’s income tax return. If such Shareholder fails to notify the Company of such inconsistent reporting, such Shareholder shall be liable to the Company for any losses, costs or expenses (including reasonable attorneys’ fees) arising from such inconsistent reporting, including an audit.

ARTICLE IV

Miscellaneous

SECTION 4.01. Confidentiality. Each of the parties agrees that any information furnished pursuant to this Agreement is confidential and, except as and to the extent required by law or otherwise during the course of an audit or contest or other administrative or legal proceeding, shall not be disclosed to other persons.

SECTION 4.02. Successors and Access to Information. This Agreement shall be binding upon and inure to the benefit of any successor to any of the parties, by merger, acquisition of assets or stock in the Company or otherwise, to the same extent as if the successor had been an original party to this Agreement or the relevant Shareholder for the taxable period in question, and in such event, all references herein to a party shall refer instead to the successor of such party; provided, however, that for purposes of calculating the estimated tax liability to which any payments under this Agreement would relate, the original Shareholder’s estimated tax liability shall be taken into account, but any payments in connection therewith shall be made to the successor of such original Shareholder.

SECTION 4.03. Governing Law. This Agreement shall be governed by and construed in accordance with the laws of the State of New York excluding (to the greatest extent permissible by law) any rule of law that would cause the application of the laws of any jurisdiction other than the State of New York.

SECTION 4.04. Headings. The headings in this Agreement are for convenience only and shall not be deemed for any purpose to constitute a part or to affect the interpretation of this Agreement.

SECTION 4.05. Counterparts. This Agreement may be executed simultaneously in two or more counterparts, each of which will be deemed an original, and it shall not be necessary in making proof of this Agreement to produce or account for more than one counterpart.

SECTION 4.06. Notices. Any notice or communication required or permitted to be given under this Agreement shall be in writing (including telecopy communication) and mailed, telecopied or delivered to the parties at the addresses specified in Schedule A or at such other address as one party may specify by notice to the other party. All such notices and communications shall be effective when received. Any payment required to be made under this Agreement shall be mailed or delivered to the parties at the addresses specified in Schedule A or at such other address or account as one party may specify by notice to the other party.

4

SECTION 4.07. Severability. If any provision of this Agreement is held to be unenforceable for any reason, it shall be adjusted rather than voided, if possible, in order to achieve the intent of the parties to the maximum extent practicable. In any event, all other provisions of this Agreement shall be deemed valid, binding, and enforceable to their full extent.

SECTION 4.08. Survival. This Agreement shall remain in force and be binding so long as the applicable period of assessments (including extensions) remains unexpired for any taxes contemplated by this Agreement.

SECTION 4.09. Successor Provisions. Any reference herein to any provisions of the Code or Treasury Regulations shall be deemed to include any amendments or successor provisions thereto as appropriate.

SECTION 4.10. Integration; Amendments. Except as explicitly stated herein, this Agreement embodies the entire understanding between the parties relating to its subject matter and supersedes and terminates all prior agreements and understandings among the parties with respect to such matters. No promises, covenants or representations of any kind, other than those expressly stated herein, have been made to induce any party to enter into this Agreement. This Agreement shall not be modified or terminated except by a writing duly signed by each of the parties hereto, and no waiver of any provisions of this Agreement shall be effective unless in a writing duly signed by the party sought to be bound.

SECTION 4.11. Waiver of Jury Trial. EACH PARTY HEREBY WAIVES, TO THE FULLEST EXTENT PERMITTED BY APPLICABLE LAW, ANY RIGHT IT MAY HAVE TO A TRIAL BY JURY WITH RESPECT TO ANY DISPUTE ARISING OUT OF THIS AGREEMENT. EACH PARTY (A) CERTIFIES THAT NO REPRESENTATIVE, AGENT OR ATTORNEY OF ANY OTHER PARTY HAS REPRESENTED, EXPRESSLY OR OTHERWISE, THAT SUCH OTHER PARTY WOULD NOT, IN THE EVENT OF LITIGATION, SEEK TO ENFORCE THE FOREGOING WAIVER AND (B) ACKNOWLEDGES THAT IT AND THE OTHER PARTIES HERETO HAVE BEEN INDUCED TO ENTER INTO THIS AGREEMENT BY, AMONG OTHER THINGS, THE MUTUAL WAIVERS AND CERTIFICATIONS IN THIS SECTION 4.11.

[Signature pages follow.]

5

IN WITNESS WHEREOF, each of the parties of this Agreement has executed this Agreement, or caused this Agreement to be executed by its duly authorized officer or trustee, as of the date first set forth above.

6

7

8

9

SCHEDULE A

Notices

To the Company:

The Fresh Market, Inc.

628 Green Valley Road, Suite 500

Greensboro, North Carolina 27408

Facsimile: (336) 272-1664

Attn: General Counsel

With a copy to:

Cravath, Swaine & Moore LLP

Worldwide Plaza

825 Eighth Avenue

New York, New York 10019

Facsimile No.: (212) 474-3700

Attention: Craig F. Arcella

To the Shareholders:

10

11

12