Exhibit 99.4

LETTER TO SECURITY DEALERS, COMMERCIAL BANKS, TRUST COMPANIES AND OTHER NOMINEES

$35,000,000 SHARES OF COMMON STOCK

ISSUABLE UPON EXERCISE OF SUBSCRIPTION RIGHTS

THE SUBSCRIPTION RIGHTS ARE EXERCISABLE BEFORE 5:00 P.M., EASTERN TIME, ON DECEMBER 14, 2011, SUBJECT TO EXTENSION OR EARLIER CANCELLATION IN THE COMPANY’S SOLE DISCRETION

November 23, 2011

Dear Brokers, Dealers, Custodian, Banks, and/or Other Nominees:

This letter is being distributed to brokers, dealers, custodian banks, and other nominees in connection with the Rights Offering by Taylor Capital Group, Inc. (the “Company”) to the holders of certain Eligible Securities.¹ The terms of the Rights Offering are included in the Prospectus Supplement, dated November 23, 2011, to the Prospectus, dated January 26, 2011 (together, the “Prospectus”). Capitalized terms used but not defined herein shall have the meanings assigned to them in the Prospectus.

As described in the Prospectus, holders of record of the Eligible Securities as of 5:00 p.m., Eastern Time, on November 21, 2011 (the “Record Date”), will receive, at no charge, nontransferable Subscriptions Rights, for up to an aggregate purchase price of $35,000,000, as set forth below:

Please note, however, that fractional Subscription Rights will not be issued in the Rights Offering. Rather, all fractional Subscription Rights will be eliminated by rounding down to the nearest whole Subscription Right.

Each securityholder’s Subscription Rights will be evidenced by a subscription rights certificate (the “Subscription Rights Certificate”) registered in the securityholder’s name or in the name of such securityholder’s nominee. In addition to the Basic Subscription Privilege set forth above, each Subscription Right also entitles the holder thereof – subject to the allocation described below and the limitations set forth in the Prospectus – to subscribe for additional Shares that have not been purchased by other Subscription Rights holders pursuant to their Basic Subscription Privileges, at the subscription price, if such holder has fully exercised his, her, or its Basic Subscription Privilege. Subject to the allocation process described below, the Company will attempt to honor all oversubscriptions in full if sufficient Shares are available.

¹ “Eligible Securities” as used herein means the Common Stock, Series C Preferred, Series D Preferred, Series E Preferred, Series G Preferred, and the Participating Warrants of the Company. Not all warrants issued by the Company are necessarily Participating Warrants that entitle their holders to receive Subscription Rights in the Rights Offering. Only those warrants issued by the Company on September 29, 2008, May 28, 2010, and October 21, 2010 are Participating Warrants.

Shares pursuant to the Oversubscription Privilege will be allocated pursuant to a two-step process. The maximum number of Shares purchasable by a securityholder in the first step of the allocation process will be limited to the result of (x) the total number of unsubscribed Shares multiplied by (y) the number of Subscription Rights that were distributed to such securityholder divided by (z) the number of Subscription Rights distributed to all holders of Eligible Securities as of the Record Date. Securityholders whose oversubscription requests exceed the Share limitation of the first step of the allocation process will proceed to the second step, where we will seek to honor oversubscription requests in full if sufficient Shares are available. If oversubscription requests for Shares in the second step of the allocation process exceed the number of Shares available, however, we will allocate the available Shares pro rata among the securityholders participating in the second step of the allocation process in proportion to the number of Subscription Rights that were distributed to each of those securityholders relative to the number of Subscription Rights distributed to all holders of Eligible Securities participating in the second step of the allocation, with the number of Shares rounded up or down, as applicable, to result in the allocation of whole Shares. If this pro rata allocation results in any securityholder receiving a greater number of Shares than the securityholder subscribed for pursuant to the exercise of the Oversubscription Privilege, then such securityholder will be allocated only that number of Shares for which the securityholder oversubscribed, and the remaining Shares will be allocated among all other securityholders exercising the Oversubscription Privilege on the same pro rata basis described above. The proration process will be repeated until all Shares have been allocated.

The Company will not issue fractional Shares. Fractional Shares resulting from the exercise of Basic Subscription Privileges and the Oversubscription Privilege will be eliminated, with the total subscription payment being adjusted accordingly. Any excess subscription payments received will be returned, without interest or deduction, as soon as practicable after the expiration of the subscription period.

If you exercise Subscription Rights on behalf of beneficial owners, you will be required to certify to the Company and Computershare Trust Company, N.A. (the “Subscription Agent”), in connection with such exercise, as to the aggregate number of Shares being subscribed for, whether the Basic Subscription Privilege of each beneficial owner of Subscription Rights on whose behalf you are acting has been exercised in full, and the number of Shares being subscribed for pursuant to the Oversubscription Privilege by each beneficial owner of Subscription Rights on whose behalf you are acting.

In connection with the Rights Offering, we are asking you to contact your clients for whom you hold Eligible Securities registered in your name or in the name of your nominee to obtain instructions with respect to the Subscription Rights.

Enclosed are copies of the following documents for your use:

2

Your prompt action is requested. The Rights Offering will expire at 5:00 p.m., Eastern Time, on December 14, 2011, subject to extension or earlier cancellation in the Company’s sole discretion (such date and time, as may be extended or cancelled, the “Expiration Date”).

To exercise the Subscription Rights, properly completed and executed Subscription Rights Certificates and payment for all Subscription Rights exercised must be delivered to the Subscription Agent as indicated in the Prospectus prior to the Expiration Date. Payment must be made to the Subscription Agent in full, in U.S. currency, and in immediately available funds, by: (i) bank check or bank draft payable to “Computer Share Trust Company, N.A., acting as Subscription Agent for Taylor Capital Group, Inc.”; or (ii) postal, telegraphic or express money order payable to “Computershare Trust Company, N.A., acting as Subscription Agent for Taylor Capital Group, Inc.”

If you have other questions regarding the Rights Offering or need additional copies of the documents relating to the Rights Offering, please contact the information agent, Georgeson, Inc., at (866) 828-4304, or, for banks and brokers, at (212) 440-9800.

Very truly yours,

Taylor Capital Group, Inc.

NOTHING HEREIN OR IN THE ENCLOSED DOCUMENTS SHALL CONSTITUTE YOU OR ANY PERSON AS AN AGENT OF TAYLOR CAPITAL GROUP, INC., THE SUBSCRIPTION AGENT, THE INFORMATION AGENT, OR ANY OTHER PERSON MAKING OR DEEMED TO BE MAKING OFFERS OF THE SHARES ISSUABLE UPON VALID EXERCISE OF THE SUBSCRIPTION RIGHTS, OR AUTHORIZE YOU OR ANY OTHER PERSON TO MAKE ANY STATEMENTS ON BEHALF OF ANY OF THEM WITH RESPECT TO THE RIGHTS OFFERING EXCEPT FOR STATEMENTS MADE IN THE PROSPECTUS.

3