Exhibit 99.6

SWAP COUNTERPARTY RIGHTS AGREEMENT

AMONG

THE ROYAL BANK OF SCOTLAND PLC,

AS PRIMARY SWAP COUNTERPARTY

ALLY AUTO RECEIVABLES TRUST 2010-3,

AS ISSUING ENTITY

ALLY FINANCIAL INC.,

AS SERVICER, CUSTODIAN, AND ADMINISTRATOR

ALLY AUTO ASSETS LLC,

AS DEPOSITOR

DEUTSCHE BANK TRUST COMPANY AMERICAS,

AS INDENTURE TRUSTEE AND PAYING AGENT

AND

BNY MELLON TRUST OF DELAWARE,

AS OWNER TRUSTEE

DATED AS OF AUGUST 18, 2010

TABLE OF CONTENTS

i

ii

THIS SWAP COUNTERPARTY RIGHTS AGREEMENT, dated as of August 18, 2010 (this “Agreement”), is among THE ROYAL BANK OF SCOTLAND PLC, as Swap Counterparty (the “Primary Swap Counterparty”), ALLY AUTO RECEIVABLES TRUST 2010-3, a Delaware statutory trust (the “Trust” or the “Issuing Entity”), ALLY FINANCIAL INC., a Delaware corporation, as Seller, Servicer, Custodian, and Administrator (“Ally Financial”), ALLY AUTO ASSETS LLC, a Delaware limited liability company, as Depositor (“Ally Auto”), DEUTSCHE BANK TRUST COMPANY AMERICAS, not in its individual capacity but solely as Indenture Trustee (the “Indenture Trustee”) and as Paying Agent (the “Paying Agent”), and BNY MELLON TRUST OF DELAWARE, a Delaware banking corporation, not in its individual capacity but solely as Owner Trustee (the “Owner Trustee”).

WHEREAS, as of the date hereof, the Trust has entered into the Interest Rate Swap with the Primary Swap Counterparty;

WHEREAS, the parties intend in this Agreement to enumerate certain rights of the Primary Swap Counterparty.

NOW, THEREFORE, in consideration of the premises and the mutual covenants herein contained, the parties hereto agree as follows:

ARTICLE I

DEFINITIONS

Section 1.01 Definitions. Capitalized terms used and not otherwise defined in this Agreement are defined in and shall have the respective meanings assigned them in Part I of Appendix A to the Trust Sale and Servicing Agreement dated as of the date hereof, among Ally Financial, as Servicer, Ally Auto, as Depositor, and the Trust, as Issuing Entity, as it may be amended, supplemented or modified from time to time (the “Trust Sale and Servicing Agreement”), and all references herein to Articles and Sections are to Articles or Sections of other Basic Documents unless otherwise specified. The rules of construction set forth in Part II of Appendix A to the Trust Sale and Servicing Agreement shall be applicable to this Agreement.

ARTICLE II

LIMITATIONS ON ISSUING ENTITY’S CONSOLIDATION,

MERGER OR SALE OF ASSETS; LIMITATIONS ON PAYMENT TO NOTEHOLDERS OR

CERTIFICATEHOLDERS

Section 2.01 Consolidation or Merger; Sale or Disposition of Assets or Property.

(a) The Issuing Entity shall not consolidate or merge with or into any other Person, unless the Issuing Entity shall have delivered to the Indenture Trustee an Officer’s Certificate and an Opinion of Counsel addressed to the Issuing Entity, each stating that such consolidation or merger and related supplemental indenture shall have no material adverse tax consequence to the Primary Swap Counterparty.

(b) Except as otherwise expressly permitted by the Indenture or the other Basic Documents, the Issuing Entity shall not sell, convey, exchange, transfer or otherwise

dispose of any of its properties or assets, including those included in the Trust Estate, to any Person, unless the Issuing Entity shall have delivered to the Indenture Trustee and the Primary Swap Counterparty an Officer’s Certificate and an Opinion of Counsel addressed to the Issuing Entity and the Primary Swap Counterparty, each stating that such sale, conveyance, exchange, transfer or disposition and related supplemental indenture shall have no material adverse tax consequence to the Primary Swap Counterparty.

Section 2.02 Limitations on Payment to Noteholders or Certificateholders. At any time that the Trust, the Indenture Trustee or any other Person under any Basic Document shall make any payment or distribution to any Noteholder or Certificateholder out of the funds in the Collection Account or out of funds realized upon any sale or liquidation (in whole or in part) of the Collateral pursuant to Article V of the Indenture, such payment (an “Investor Payment”) shall not be made unless the Primary Swap Counterparty has received any and all payments that it is entitled to receive pursuant to Sections 4.06(c)(ii) and (iii) and 8.01(b) of the Trust Sale and Servicing Agreement (or funds have been set aside for the purpose of making such payments) in priority to such Investor Payment or on a parity with such Investor Payment pursuant to the terms of such provisions.

ARTICLE III

AMENDMENTS TO TRANSACTION DOCUMENTS

Section 3.01 Amendments to the Trust Sale and Servicing Agreement. The Trust Sale and Servicing Agreement may be amended, modified or supplemented from time to time by the Depositor, the Servicer and the Issuing Entity in the manner specified by Section 9.01(b) of the Trust Sale and Servicing Agreement; provided, however, that no such amendment, modification or supplement shall be made unless either (A) the Primary Swap Counterparty consents in writing to such amendment, modification or supplement or (B) the amendment, modification or supplement will, as evidenced by an Opinion of Counsel addressed to the Primary Swap Counterparty, have no material adverse effect on the interests of the Primary Swap Counterparty.

Section 3.02 Amendments to Trust Agreement. The Trust Agreement may be amended, modified or supplemented from time to time by the Depositor and the Owner Trustee in the manner specified by Section 8.2 of the Trust Agreement; provided, however, that no amendment, modification or supplement shall be made unless either (A) the Primary Swap Counterparty consents in writing to such amendment, modification or supplement or (B) the amendment, modification or supplement will, as evidenced by an Opinion of Counsel addressed to the Primary Swap Counterparty, have no material adverse effect on the interests of the Primary Swap Counterparty.

Section 3.03 Amendments to Administration Agreement. The Administration Agreement may be amended, modified or supplemented from time to time pursuant to Section 13 of the Administration Agreement; provided, however, that no amendment, modification or supplement will be effected pursuant to Section 13 of the Administration Agreement unless either (A) the Primary Swap Counterparty consents in writing to such amendment, modification or supplement or (B) such amendment, modification or supplement will, as evidenced by an Opinion of Counsel addressed to the Primary Swap Counterparty, have no material adverse effect on the interests of the Primary Swap Counterparty.

2

Section 3.04 Amendments to Custodian Agreement. The Custodian Agreement may be amended, modified or supplemented pursuant to Section 9 thereof; provided, however, that no such amendment, modification or supplement shall be made unless either (A) the Primary Swap Counterparty consents in writing to such amendment, modification or supplement or (B) such amendment, modification or supplement will, as evidenced by an Opinion of Counsel addressed to the Primary Swap Counterparty, have no material adverse effect on the interests of the Primary Swap Counterparty.

Section 3.05 Supplemental Indentures. No supplemental indenture shall be entered into under Section 9.1(b) or 9.2 of the Indenture unless either (A) the Primary Swap Counterparty consents in writing to such supplemental indenture or (B) such supplemental indenture will, as evidenced by an Opinion of Counsel addressed to the Primary Swap Counterparty, have no material adverse effect on the interests of the Primary Swap Counterparty.

ARTICLE IV

DELIVERY OF NOTICES AND REPORTS

Section 4.01 Notices of Replacement of Indenture Trustee.

(a) The Indenture Trustee shall provide the Primary Swap Counterparty with a copy of any notice of its intent to resign delivered pursuant to Section 6.8(a) of the Indenture.

(b) A successor Indenture Trustee shall deliver to the Primary Swap Counterparty a copy of any acceptance under Section 6.8(c) of the Indenture.

Section 4.02 Notices of Events of Default, Enforcement and Termination.

(a) The Paying Agent shall give the Primary Swap Counterparty notice of any default by the Issuing Entity (or any other obligor upon the Notes) of which it has actual knowledge in the making of any payment required to be made with respect to the Notes.

(b) The Issuing Entity shall deliver to the Primary Swap Counterparty a copy of any notice it shall deliver pursuant to Section 3.7(d) of the Indenture in respect of the occurrence of a Servicer Default under the Trust Sale and Servicing Agreement.

(c) The Issuing Entity shall give the Primary Swap Counterparty prompt written notice of each Event of Default under the Indenture, each Servicer Default, each default on the part of the Depositor of its obligations under the Trust Sale and Servicing Agreement and each default on the part of the Seller of its obligations under the Pooling and Servicing Agreement.

3

(d) The Issuing Entity shall deliver to the Primary Swap Counterparty, within five Business Days after learning of the occurrence thereof, a copy of the written notice in the form of an Officer’s Certificate delivered to the Indenture Trustee, of any event which with the giving of notice and the lapse of time would become an Event of Default under Section 5.1(d) of the Indenture, its status and what action the Issuing Entity is taking or proposes to take with respect thereto.

(e) If an Event of Default should occur and be continuing under the Indenture, and the Indenture Trustee or the requisite percentage of the Holders of the Notes declare all of the Notes immediately due and payable pursuant to Section 5.2(a) of the Indenture, then the Indenture Trustee shall give prompt written notice thereof to the Primary Swap Counterparty.

(f) The Indenture Trustee shall promptly give to the Primary Swap Counterparty written notice of any waiver pursuant to Section 5.12 of the Indenture.

(g) The Indenture Trustee shall promptly provide to the Primary Swap Counterparty written notice of each request for action that is made and direction received pursuant to Section 5.16 of the Indenture, with respect to the exercise of the Indenture Trustee’s powers to compel performance or enforce the obligations of the parties under the Basic Documents.

(h) The Indenture Trustee shall mail to the Primary Swap Counterparty any notice of Default which the Indenture Trustee mails to Noteholders pursuant to Section 6.5 of the Indenture.

(i) The Servicer shall deliver to the Primary Swap Counterparty promptly after having obtained knowledge thereof, but in no event later than five Business Days thereafter, written notice in an officer’s certificate of any event which with the giving of notice or lapse of time, or both, would become a Servicer Default under Section 7.01 of the Trust Sale and Servicing Agreement.

(j) Notice of any termination of the Trust shall be given by the Servicer to the Primary Swap Counterparty as soon as practicable after the Servicer has received notice thereof.

Section 4.03 Notices of Amendment of the Custodian Agreement. Promptly after the execution of an amendment to, or consent under, the Custodian Agreement, the Custodian shall furnish written notification of the substance of such amendment or consent to the Primary Swap Counterparty.

Section 4.04 Notices of Amendment of the Administration Agreement. Promptly after the execution of an amendment to, or consent under, the Administration Agreement, the Administrator shall furnish written notification of the substance of such amendment or consent to the Primary Swap Counterparty.

4

Section 4.05 Notices of Supplemental Indentures. Promptly after the execution of an indenture supplemental to, or consent under, the Indenture, the Issuing Entity shall furnish written notification of the substance of such amendment or consent to the Primary Swap Counterparty.

Section 4.06 Notices of Amendment of the Trust Agreement. Promptly after the execution of an amendment to, or consent under, the Trust Agreement, Ally Auto shall furnish written notification of the substance of such amendment or consent to the Primary Swap Counterparty.

Section 4.07 Notices of Amendment of the Trust Sale and Servicing Agreement. Promptly after the execution of an amendment to, or consent under, the Trust Sale and Servicing Agreement, Ally Auto shall furnish written notification of the substance of such amendment or consent to the Primary Swap Counterparty.

Section 4.08 Notices of Release of Property. The Indenture Trustee shall provide copies to the Primary Swap Counterparty of all of the documents received by it pursuant to Section 8.4(b) of the Indenture.

Section 4.09 Notices of Release of Collateral. The Indenture Trustee shall provide copies to the Primary Swap Counterparty of any document received by it pursuant to Section 2.9 of the Indenture with respect to the release of Collateral.

Section 4.10 Notices of Removal of Administrator.

(a) Subject to Section 10(e) of the Administration Agreement, the Issuing Entity may remove the Administrator without cause pursuant to Section 10(c) of the Administration Agreement by providing the Administrator and the Primary Swap Counterparty with at least 60 days’ prior written notice.

(b) The Issuing Entity shall provide to the Primary Swap Counterparty a copy of any written notice from the Issuing Entity to the Administrator effecting the immediate removal of the Administrator pursuant to Section 10(d) of the Administration Agreement.

Section 4.11 Notices of Assignment of the Trust Sale and Servicing Agreement. The Depositor shall provide to the Primary Swap Counterparty notice of any assignment of the Trust Sale and Servicing Agreement made pursuant to Section 9.06 thereof.

Section 4.12 Notices Generally. The Issuing Entity shall promptly transmit any notice received by it from the Noteholder(s) to the Primary Swap Counterparty. The Indenture Trustee shall likewise promptly transmit any notice received by it from such Noteholder(s) to the Primary Swap Counterparty.

5

Section 4.13 Delivery of Reports.

(a) The Servicer shall cause the firm delivering the Report of Assessment of Compliance with Servicing Criteria pursuant to Section 4.02(a) of the Trust Sale and Servicing Agreement, to deliver a copy thereof to the Primary Swap Counterparty.

(b) The Issuing Entity shall deliver to the Primary Swap Counterparty a copy of the Annual Statement of Compliance required by Section 3.9 of the Indenture.

(c) On each Determination Date, the Servicer shall deliver to the Primary Swap Counterparty a copy of the Servicer’s Accounting required by Section 3.10 of the Pooling and Servicing Agreement.

(d) The Servicer shall deliver to the Primary Swap Counterparty, promptly after the execution and delivery of the Trust Sale and Servicing Agreement and of each amendment thereto, an Opinion of Counsel as required in Section 9.02 (j) thereof.

Section 4.14 Delivery Time. Except as otherwise specified in this Article IV, any delivery of a notice, report or any other document required under this Article IV shall be made within five Business Days after (i) such notice, report or document becomes available to the Person who is required to make such delivery to the Primary Swap Counterparty or (ii) the specified event with respect to such notice occurs and is known to such Person, as applicable.

Section 4.15 Notice of Redemption. The Issuing Entity or the Servicer shall deliver to the Primary Swap Counterparty a copy of any notice of redemption of the Notes that the Servicer shall furnish to the Indenture Trustee pursuant to Section 10.1 of the Indenture.

ARTICLE V

MISCELLANEOUS

Section 5.01 Notices. All demands upon, notices to and communications with the Primary Swap Counterparty required hereunder shall be delivered in the manner specified for notices in the Interest Rate Swaps, and all other demands upon, notices to and communications upon or to the other parties hereto shall be delivered as specified in Appendix B of the Trust Sale and Servicing Agreement.

Section 5.02 GOVERNING LAW. THIS AGREEMENT SHALL BE GOVERNED BY, AND CONSTRUED IN ACCORDANCE WITH, THE LAWS OF THE STATE OF NEW YORK WITHOUT REFERENCE TO ITS CONFLICT OF LAW PROVISIONS BUT WITHOUT PREJUDICE TO THE PROVISIONS OF SECTIONS 5-1401 AND 5-1402 OF ITS

6

GENERAL OBLIGATIONS LAW, AND THE OBLIGATIONS, RIGHTS AND REMEDIES OF THE PARTIES HEREUNDER SHALL BE DETERMINED IN ACCORDANCE WITH SUCH LAWS.

Section 5.03 Binding Effect. This Agreement shall be binding upon and shall inure to the benefit of all of the parties hereto and their respective successors and assigns, including the Issuing Entity.

Section 5.04 Replacement of the Swap Counterparty. In the event that the Primary Swap Counterparty resigns, is removed or otherwise replaced as Primary Swap Counterparty pursuant to the terms of the Interest Rate Swaps, its assignee or successor in interest thereunder shall, subject to satisfying the Rating Agency Condition, automatically succeed to the interests of the Primary Swap Counterparty under this Agreement.

Section 5.05 Severability of Provisions. If any one or more of the covenants, agreements, provisions or terms of this Agreement shall be for any reason whatsoever held invalid, then such covenants, agreements, provisions or terms shall be deemed severable from the remaining covenants, agreements, provisions or terms of this Agreement and shall in no way affect the validity or enforceability of the other provisions of this Agreement.

Section 5.06 Assignment. This Agreement may not be assigned by the Primary Swap Counterparty without the prior written consent of each of the Trust, the Indenture Trustee, the Owner Trustee, Ally Financial, and Ally Auto, except as provided in Section 5.04 hereof.

Section 5.07 Amendments. No change or amendment to this Agreement will be effective unless in writing and signed by all of the parties to this Agreement.

Section 5.08 Headings. The headings of the various Sections herein are for convenience of reference only and shall not define or limit any of the terms or provisions hereof.

Section 5.09 Counterparts. This Agreement may be executed by the parties in separate counterparts (including by facsimile transmission), each of which when so executed and delivered shall be an original but all such counterparts shall together constitute but one and the same instrument.

Section 5.10 Limitation of Liability. It is expressly understood and agreed by the parties hereto that (a) this Agreement is executed and delivered by BNY Mellon Trust of Delaware, not in its individual capacity but solely as Owner Trustee of the Issuing Entity, in the exercise of the powers and authority conferred and vested in it under the Trust Agreement, (b) each of the representations, undertakings and agreements herein made on the part of the Issuing Entity or the Owner Trustee is made and intended not as personal representations, undertakings

7

and agreements by BNY Mellon Trust of Delaware but is made and intended for the purpose of binding only the Issuing Entity and (c) under no circumstances shall BNY Mellon Trust of Delaware be personally liable for the payment of any indebtedness or expenses of the Issuing Entity or the Owner Trustee or be liable for the breach or failure of any obligation, representation, warranty or covenant made or undertaken by the Issuing Entity or the Owner Trustee under this Agreement or the other Basic Documents. For all purposes of this Agreement, in the performance of any duties or obligations of the Issuing Entity or the Owner Trustee hereunder, BNY Mellon Trust of Delaware shall be entitled to the benefits of the terms and provisions of the Trust Agreement.

Section 5.11 Termination. This Agreement shall terminate upon termination of the Interest Rate Swap and payment in full of all amounts owing to the Primary Swap Counterparty in respect thereof.

Section 5.12 Agency Role of RBS Securities Inc. In connection with this Agreement, RBS Securities Inc. has acted as agent on behalf of the Primary Swap Counterparty. RBS Securities Inc. has not guaranteed and is not otherwise responsible for the obligations of the Primary Swap Counterparty under this Agreement.

Section 5.13 Rights of Indenture Trustee and Paying Agent. For all purposes of this Agreement, in the performance of its duties or obligations hereunder or in the performance of any duties or obligations of the Issuing Entity hereunder, the Indenture Trustee shall be subject to, and entitled to the benefits of, the terms and provisions of Article VI of the Indenture.

*    *    *    *

8

IN WITNESS WHEREOF, the parties hereto have caused this Primary Swap Counterparty Rights Agreement to be duly executed by their respective officers, thereunto duly authorized, all as of the day and year first above written.