Exhibit 10.3

FIRST AMENDMENT TO CREDIT AGREEMENT AND FIRST AMENDMENT TO COLLATERAL AGREEMENT

FIRST AMENDMENT TO CREDIT AGREEMENT AND FIRST AMENDMENT TO COLLATERAL AGREEMENT, dated as of February 26, 2013 (collectively, this “First Amendment”), among OLLIE’S HOLDINGS, INC., a Delaware corporation (“Ollie’s Holdings”), OLLIE’S BARGAIN OUTLET, INC., a Pennsylvania corporation (“Ollie’s” and, together with Ollie’s Holdings, the “Borrowers”), BARGAIN PARENT, INC., a Delaware corporation (“Parent”), the Lenders (as defined in the Credit Agreement referred to below) party hereto and MANUFACTURERS AND TRADERS TRUST COMPANY, as administrative agent for the Lenders (in such capacity, the “Administrative Agent”). All capitalized terms used herein and not otherwise defined herein shall have the respective meanings provided to such terms in the Credit Agreement referred to below.

W I T N E S S E T H:

WHEREAS, Parent, the Borrowers, the Lenders, the Administrative Agent and the other parties thereto are parties to that certain Credit Agreement, dated as of September 28, 2012 (the “Credit Agreement”);

WHEREAS, Parent, the Borrowers, the Administrative Agent and each Lender party hereto desire to amend the Credit Agreement to permit Ollie’s and Ollie’s Holdings to pay the Special Dividend (as defined below) to Parent and to make certain other changes to the Credit Agreement and the Collateral Agreement, in each case, as provided herein; and

WHEREAS, concurrently with the effectiveness of this First Amendment, Parent, the Borrowers, the Lenders party thereto and the Term Agent intend to amend (the “Term Facility Amendment”) the Term Facility to make certain changes to the Term Facility, including to permit the payment of the Special Dividend.

NOW, THEREFORE, for good and valuable consideration, the receipt and adequacy of which is acknowledged by each party hereto, it is agreed:

A. Section 1.01 of the Credit Agreement is hereby amended as follows:

1. the definition of “Secured Bank Product Obligations” is hereby amended by inserting the following proviso immediately before the period appearing at the end of such definition:

“; provided that in no event shall an Excluded Swap Obligation constitute a Secured Bank Product Obligation.”

2. the definition of “Secured Obligations” is hereby amended by inserting the following parenthetical “(other than Excluded Swap Obligations)” immediately after the text “Secured Bank Product Obligations” contained therein; and

3. by inserting in the appropriate alphabetical order the following new definitions:

“Commodity Exchange Act” shall mean the Commodity Exchange Act (7 U.S.C. § 1 et seq.).

“ECP” shall have the meaning assigned to such term in the definition of Excluded Swap Obligation.

“Excluded Swap Obligation” shall mean, with respect to any Loan Party, any Swap Obligation if, and to the extent that, all or a portion of the Guarantee of such Loan Party of, or the grant by such Loan Party of a security interest to secure, such Swap Obligation (or any Guarantee thereof) is or becomes illegal under the Commodity Exchange Act or any rule, regulation or order of the Commodity Futures Trading Commission (or the application or official interpretation of any thereof) by virtue of such Loan Party’s failure for any reason to constitute an “eligible contract participant” as defined in the Commodity Exchange Act and the regulations thereunder (each an “ECP”) at the time the Guarantee of such Loan Party or the grant of such security interest becomes effective with respect to such Swap Obligation. If a Swap Obligation arises under a master agreement governing more than one swap, such exclusion shall apply only to the portion of such Swap Obligation that is attributable to swaps for which such Guaranty or security interest is or becomes illegal.

“First Amendment” shall mean the First Amendment to Credit Agreement and First Amendment to Collateral Agreement, dated as of February 26, 2013, among Parent, the Borrowers, the Lenders party thereto and the Administrative Agent.

“First Amendment Effective Date” shall mean the date on which the First Amendment is effective pursuant to the terms thereof.

“Qualified ECP Guarantor” shall mean, in respect of any Swap Obligation, each Loan Party that has total assets exceeding $10,000,000 at the time the relevant Guaranty or grant of the relevant security interest becomes effective with respect to such Swap Obligation is incurred or such other person as constitutes an ECP under the Commodity Exchange Act or any regulations promulgated thereunder and can cause another person to qualify as an ECP at such time by entering into a keepwell under Section 1(a)(18)(A)(v)(II) of the Commodity Exchange Act.

“Special Dividend” shall have the meaning assigned to such term in Section 6.06(m).

“Swap Obligation” shall mean, with respect to any Loan Party, any obligation to pay or perform under any Swap Agreement to the extent that such Swap Agreement constitutes a “swap” within the meaning of Section 1a(47) of the Commodity Exchange Act.

-2-

B. Section 6.06 of the Credit Agreement is hereby amended by (i) deleting the word “and” appearing at the end of clause (k) thereof, (ii) deleting the period appearing at the end of clause (l) thereof and inserting “; and” in lieu thereof and (iii) adding the following new clause (m) thereto:

“(m) as soon as reasonably practicable after the First Amendment Effective Date, Ollie’s may use the proceeds of the Incremental Term Loans to pay a cash dividend to Ollie’s Holdings which in turn may use the proceeds of the Incremental Term Loans and/or the proceeds received from Ollie’s as provided herein to pay a cash dividend to Parent in an aggregate amount not to exceed $46,200,000 (the “Special Dividend”).”

A. The definition of “Secured Obligations” appearing in Section 1.1(b) of the Collateral Agreement is hereby amended by inserting the following sentence at the end thereof:

“Notwithstanding anything to the contrary contained herein or in any other Loan Document, in no event will the Secured Obligations include any Excluded Swap Obligations.”

B. The Collateral Agreement is hereby further amended by inserting the following new Section 2.8 immediately after Section 2.7 thereof:

“2.8. Keepwell. Each Qualified ECP Guarantor hereby jointly and severally absolutely, unconditionally and irrevocably undertakes to provide such funds or other support as may be needed from time to time by each other Guarantor to honor all of its obligations under the guarantee contained in this Section 2 in respect of Swap Obligations (provided, however, that each Qualified ECP Guarantor shall only be liable under this Section 2.8 for the maximum amount of such liability that can be hereby incurred without rendering its obligations under this Section 2.8, or otherwise under the guarantee contained in this Section 2, voidable under applicable law relating to fraudulent conveyance or fraudulent transfer, and not for any greater amount). The obligations of each Qualified ECP Guarantor under this Section 2.8 shall remain in full force and effect until the Termination Date. Each Qualified ECP Guarantor intends that this Section 2.8 constitute, and this Section 2.8 shall be deemed to constitute, a “keepwell, support, or other agreement” for the benefit of each other Guarantor for all purposes of Section 1a(18)(A)(v)(II) of the Commodity Exchange Act.”

In order to induce the Administrative Agent and the undersigned Lenders to enter into this First Amendment, Parent and each Borrower hereby represent and warrant that:

A. no Default or Event of Default has occurred and is continuing on the date hereof or will have occurred and be continuing as of the First Amendment Effective Date, both immediately before and immediately after giving effect to the First Amendment;

B. all of the representations and warranties of each Loan Party contained in the Credit Agreement and in the other Loan Documents are true and correct in all material respects (without duplication of any materiality qualifier contained therein) on the date hereof and will be

-3-

true and correct in all material respects (without duplication of any materiality qualifier contained therein) on the First Amendment Effective Date, both immediately before and immediately after giving effect to the First Amendment, as though made on and as of the First Amendment Effective Date (except for representations and warranties that expressly relate to an earlier date, in which case such representations and warranties shall be true and correct in all material respects (without duplication of any materiality qualifier contained therein) as of such earlier date);

C. (x) this First Amendment has been duly authorized by all corporate, stockholder, limited partnership or limited liability company action required to be obtained by each Loan Party, (y) this First Amendment has been duly executed and delivered by each Loan Party and (z) this First Amendment, and the Credit Agreement (as amended by this First Amendment), each constitutes a legal, valid and binding obligation of each Loan Party, enforceable against such Loan Party in accordance with its respective terms, subject to (i) the effects of bankruptcy, insolvency, moratorium, reorganization, fraudulent conveyance or other similar laws affecting creditors’ rights generally, (ii) general principles of equity (regardless of whether such enforceability is considered in a proceeding in equity or at law) and (iii) implied covenants of good faith and fair dealing; and

D. the execution, delivery and performance by each Loan Party of this First Amendment does not and will not (i) violate (A) any provision of (x) law, statute, rule or regulation applicable to such Loan Party, or (y) of the certificate or articles of incorporation or other constitutive documents or by-laws of such Loan Party, (B) any applicable order of any court or any rule, regulation or order of any Governmental Authority or (C) any provision of any indenture, certificate of designation for preferred stock, agreement or other instrument to which any Loan Party is a party or by which any of them or any of their property is or may be bound, (ii) be in conflict with, result in a breach of or constitute (alone or with notice or lapse of time or both) a default under, give rise to a right of or result in any cancellation or acceleration of any right or obligation (including any payment) or to a loss of a material benefit under any such indenture, certificate of designation for preferred stock, agreement or other instrument, where any such conflict, violation, breach or default referred to in clause (i)(A)(x), (i)(B), (i)(C) or (ii) of this paragraph, could reasonably be expected to have, individually or in the aggregate, a Material Adverse Effect, or (iii) result in the creation or imposition of any Lien upon or with respect to any property or assets now owned or hereafter acquired by any Loan Party, other than the Liens created by the Loan Documents and Liens permitted by Section 6.02 of the Credit Agreement or where such creation or imposition of any Lien could not reasonably be expected to have, individually or in the aggregate, a Material Adverse Effect.

Upon satisfaction of each of the following conditions, this First Amendment shall be deemed effective (such date, the “First Amendment Effective Date”):

A. Parent, each Borrower, each other Loan Party (if any), the Administrative Agent, and the Required Lenders, shall have signed a counterpart hereof and shall have delivered (including by way of facsimile or other electronic transmission) the same to the Administrative Agent, c/o White & Case LLP, 1155 Avenue of the Americas, New York, NY 10036 Attention: Joseph Capasso (facsimile number: 212-354-8113 / e-mail address: joseph.capasso@whitecase.com) or at such other address as shall be reasonably acceptable to the Administrative Agent;

-4-

B. all fees and expenses required to be paid to the Administrative Agent on the First Amendment Effective Date (including, without limitation, reasonable legal fees and expenses) shall have been paid;

C. no Default or Event of Default shall have occurred and be continuing or would occur after giving effect to this First Amendment;

D. all of the representations and warranties of each Loan Party contained in this First Amendment, the Credit Agreement and in the other Loan Documents are true and correct in all material respects (without duplication of any materiality qualifier contained therein) on and as of the First Amendment Effective Date, both immediately before and immediately after giving effect to the First Amendment, as though made on and as of the First Amendment Effective Date (except for representations and warranties that expressly relate to an earlier date, in which case such representations and warranties shall be true and correct in all material respects (without duplication of any materiality qualifier contained therein) as of such earlier date);

E. the First Amendment Effective Date (as defined in the Term Facility Amendment) shall have occurred; and

F. the Administrative Agent shall have received a certificate, dated the First Amendment Effective Date and signed by a Responsible Officer of Parent or the Lead Borrower, certifying on behalf of Parent and each Borrower that the conditions in this Section IV. (C) and (D) have been satisfied.

A. This First Amendment is limited to the matters specified herein and shall not constitute a modification, acceptance or waiver of any other provision of the Credit Agreement or any other Loan Document and each of the Loan Documents as modified hereby shall remain in full force and effect in accordance with their respective terms.

B. This First Amendment may be executed in any number of counterparts and by different parties on separate counterparts, each of which, when executed and delivered, shall be deemed to be an original, and all of which, when taken together, shall constitute but one and the same agreement. Delivery of an executed counterpart of this First Amendment by telefacsimile or other electronic method of transmission shall be equally as effective as delivery of an original executed counterpart of this First Amendment.

C. THE VALIDITY OF THIS FIRST AMENDMENT, THE CONSTRUCTION, INTERPRETATION, AND ENFORCEMENT HEREOF, AND THE RIGHTS OF THE PARTIES HERETO WITH RESPECT TO ALL MATTERS ARISING HEREUNDER OR RELATED HERETO (WHETHER IN CONTRACT, TORT OR OTHERWISE) SHALL BE DETERMINED UNDER, GOVERNED BY, AND CONSTRUED IN ACCORDANCE WITH THE LAWS OF THE STATE OF NEW YORK (WITHOUT REGARD TO ANY CHOICE OF LAW PRINCIPLES THAT REQUIRE THE APPLICATION OF THE LAWS OF ANOTHER JURISDICTION).

-5-

D. From and after the First Amendment Effective Date, all references in the Credit Agreement and each of the other Loan Documents to “this Agreement”, “hereunder”, “hereof”, “herein”, or words of like import and each reference to the Credit Agreement and the Collateral Agreement shall be deemed to be references to the Credit Agreement and the Collateral Agreement, as applicable, as modified hereby on the First Amendment Effective Date. This First Amendment shall constitute a “Loan Document” for all purposes under the Credit Agreement and the other Loan Documents.

*        *        *

-6-

IN WITNESS WHEREOF, the parties hereto have caused their duly authorized officers to execute and deliver this First Amendment as of the date first above written.

[Signature Page to Ollie’s ABL Credit Agreement First Amendment]

[Signature Page to Ollie’s ABL Credit Agreement First Amendment]

[Signature Page to Ollie’s ABL Credit Agreement First Amendment]

[Signature Page to Ollie’s ABL Credit Agreement First Amendment]