UNITED STATES
SECURITIES AND EXCHANGE COMMISSION

Form 10-K

ANNUAL REPORT PURSUANT TO SECTION 13 OR 15(d)
OF THE SECURITIES EXCHANGE ACT OF 1934

For the fiscal year ended December 31, 2003

Commission file number 0-15956

Bank of Granite Corporation

Securities registered pursuant to Section 12(b) of the Act: None

Securities registered pursuant to Section 12(g) of the Act:

Common Stock, $1.00 par value

     Indicate by check mark if disclosure of delinquent filers pursuant to Item 405 of Regulation S-K is not contained herein, and will not be contained, to the best of registrant’s knowledge, in the definitive proxy or information statements incorporated by reference in Part III of this Form 10-K or any amendment to this Form 10-K.   [  ]

     Indicate by check mark whether the registrant (1) has filed all reports required to be filed by Section 13 or 15(d) of the Securities Exchange Act of 1934 during the preceding 12 months (or for such shorter period that the registrant was required to file such reports), and (2) has been subject to such filing requirements for the past 90 days. Yes [X] No [  ]

     Indicate by check mark whether the registrant is an accelerated filer (as defined in Exchange Act Rule 12-b-2). Yes [X] No [  ]

     As of March 8, 2004, 13,606,966 shares of common stock, $1 par value, were outstanding.

     As of June 30, 2003, the aggregate market value of common stock held by non-affiliates was $205,486,732.

Documents Incorporated by Reference

     PART III: Definitive Proxy Statement dated March 19, 2004 as filed pursuant to Section 14 of the Securities Exchange Act of 1934 in connection with the 2004 Annual Meeting of Shareholders.

Exhibit Index begins on page 75

1

FORM 10-K CROSS-REFERENCE INDEX

* Exhibits, Financial Statement Schedules and Reports on Forms 8-K, included in or incorporated by reference into this filing were filed with the Securities and Exchange Commission. Bank of Granite Corporation provides these documents through its Internet site at www.bankofgranite.com or by mail upon request.

2

PART I

ITEM 1 - BUSINESS

Bank of Granite Corporation (the “Company”) is a Delaware corporation organized June 1, 1987 as a bank holding company. The Company currently engages in no operations other than ownership and operation of Bank of Granite (the “Bank”), a state bank chartered under the laws of North Carolina on August 2, 1906 and Granite Mortgage, Inc., formerly GLL & Associates, Inc., a mortgage bank chartered under the laws of North Carolina on June 24, 1985. Granite Mortgage was acquired by the Company on November 5, 1997. On July 15, 2003, the Company acquired First Commerce Corporation and its wholly owned subsidiary, First Commerce Bank (referred to herein collectively as “First Commerce”), and merged First Commerce Bank into the Bank on July 24, 2003. First Commerce Bank operated three banking offices in the Charlotte metropolitan area. The Company conducts its community banking business operations from 19 full-service offices and 1 loan-production office located in Caldwell, Catawba, Burke, Watauga, Wilkes and Mecklenburg counties in North Carolina. According to the North Carolina Commissioner of Banks, the Bank ranked 10th in assets and 10th in deposits among North Carolina commercial banks as of September 30, 2003. The Company conducts its mortgage banking business operations from 11 offices in the Central and Southern Piedmont and Catawba Valley regions of North Carolina and in Hilton Head Island, South Carolina.

The Company conducts its business through three reportable business segments: Community Banking, Mortgage Banking and Other. The Community Banking segment offers a variety of loan and deposit products and other financial services. The Mortgage Banking segment originates, retains and sells mortgage loans. The Other segment includes activities at the holding company level such as corporate and shareholder relations and funding from the issuance of commercial paper and trust preferred securities. For financial information on the Company’s three business segments, see Note 21 of the “Notes to Consolidated Financial Statements” on page 67.

GENERAL BUSINESS

The Bank’s principal community banking activities include the taking of demand and time deposits and the making of loans, secured and unsecured, to individuals, associations, partnerships and corporations. The Bank is an independent community bank. The majority of its customers are individuals and small businesses. No material part of its business is dependent upon a single customer or a few customers whose loss would have an adverse effect on the business of the Bank. No material portion of the business of the Bank is seasonal.

Granite Mortgage’s principal mortgage banking activities include the origination and underwriting of mortgage loans to individuals. Granite Mortgage also sells mortgage servicing rights and appraisal services. Granite Mortgage specializes in government guaranteed mortgage products. The majority of its customers are individuals. No material part of its business is dependent upon a single customer or a few customers whose loss would have an adverse effect on the business of Granite Mortgage. The mortgage business is sensitive to changes in interest rates in the market. When rates decline, Granite Mortgage experiences an increase in its mortgage business. When rates rise, Granite Mortgage’s business declines.

GENERAL DESCRIPTION OF ECONOMIC AREAS

Prior to 2003, the Company conducted its community banking operations primarily in Caldwell, Catawba and Burke counties of North Carolina. This area was historically known as a center for the manufacture of fiber optic and coaxial cable, furniture, and apparel. When the economy began to weaken in 2001, these counties were significantly impacted with a sudden and sustained rise in their unemployment rates. All of these industries faced massive layoffs of their workforces. In 2003, as opportunities arose to expand and diversify its market areas, the Company did so by entering three new markets where the local economies were better diversified and growing. The Bank opened new banking offices in Watauga (Boone) and Wilkes (Wilkesboro) counties in April and acquired First Commerce Bank and its three banking offices in Mecklenburg County (Charlotte and Cornelius) in July. The Bank also plans to open a banking office in Forsyth County

3

(Winston-Salem) in 2004. The relative unemployment rates and the population growth in Mecklenburg County, each as shown in the tables below, formed the primary basis for the Company’s decision to expand.

TERRITORY SERVED AND COMPETITION

The Bank operates community banking offices in Granite Falls and the Baton section of Granite Falls; Lenoir and the Hibriten and Whitnel sections of Lenoir; Hudson; Newton; Morganton; Hickory and the Springs Road, Viewmont, Long View and Mountain View sections of Hickory; Boone; Wilkesboro; Charlotte and the SouthPark section of Charlotte; Cornelius and Conover; for a total of 19 full-service offices, as well as a loan-production office in Matthews. In March 2003, the Bank sold its banking office in Vale, North Carolina. The Bank plans to open a banking office in Winston-Salem in 2004.

The Federal Deposit Insurance Corporation (the “FDIC”) collects deposit data from insured depository institutions as of June 30 of each year.

As of June 30, 2003, there were 8 other banks in the Bank’s Caldwell County market. Also as of June 30, 2003, the Bank had $271 million, or 35.9%, of total county deposits of $754.1 million, compared with $243.7 million, or 33.9%, of total county deposits of $719.9 million as of June 30, 2002.

According to the FDIC data, in the Bank’s Catawba County market, there were 10 other banks as of June 30, 2003. The Bank had $305.6 million, or 13.7%, of total county deposits of $2.2 billion, compared with $256.9 million, or 12%, of total county deposits of $2.1 billion as of June 30, 2002.

In the Bank’s Burke County market, there were 8 other banks as of June 30, 2003 according to the FDIC. The Bank had $38.4 million, or 5.9%, of total county deposits of $646 million, compared with $31.2 million, or 4.8%, of $650 million in total Burke County deposits as of June 30, 2002.

4

As of June 30, 2003, there were 17 other banks in the Bank’s new Mecklenburg County market. Also as of June 30, 2003, First Commerce, acquired by the Bank on July 15, 2003, had $149.2 million, or 0.3%, of total county deposits of $59.3 billion, compared with $135.8 million, or 0.4%, of total county deposits of $37.8 billion as of June 30, 2002.

In both Watauga and Wilkes counties, the Bank began accepting deposits in full-service banking offices in August 2003 and, therefore, had no deposits as of June 30, 2003.

The mortgage banking business is also highly competitive, with both bank and nonbank mortgage originators competing in the market. Granite Mortgage conducts its mortgage banking business from 10 offices in the North Carolina cities of Winston-Salem, Hickory, High Point, Lenoir, Morganton, Newton, Salisbury, Boone and Charlotte, and from one office in Hilton Head Island, South Carolina.

The Company’s Community Banking and Mortgage Banking operations are required to compete based on price in order to conduct business in each of the Company’s markets. However, the Company believes that its focus on and commitment to providing superior customer service is what distinguishes it from its competitors.

EMPLOYEES

As of December 31, 2003, the Bank had 238 and Granite Mortgage had 58 full-time equivalent employees. Each of the Bank and Granite Mortgage considers its relationship with its employees to be excellent.

SUPERVISION AND REGULATION

The following summaries of statutes and regulations affecting bank holding companies, banks and mortgage banks do not purport to be complete. Such summaries are qualified in their entirety by reference to such statutes and regulations.

The Company is a bank holding company within the meaning of the Bank Holding Company Act of 1956, as amended, and is required to register as such with the Board of Governors of the Federal Reserve System (the “Federal Reserve Board” or “FRB”).

A bank holding company is required to file with the FRB annual reports and other information regarding its business operations and those of its subsidiaries. It is also subject to examination by the Federal Reserve Board and is required to obtain Federal Reserve Board approval prior to acquiring, directly or indirectly, more than 5% of the voting stock of a bank, unless it already owns a majority of the voting stock of the bank. Furthermore, with limited exceptions, a bank holding company must engage only in the business of banking or managing or controlling banks or furnishing services to or performing services for its subsidiary banks. One of the exceptions to this prohibition is the ownership of shares of a company the activities of which the FRB has determined to be so closely related to banking or managing or controlling banks as to be a proper incident thereto.

The FRB has cease-and-desist powers over bank holding companies and non-banking subsidiaries where their action would constitute a serious threat to the safety, soundness or stability of a subsidiary bank.

Although the Company is not presently subject to any regulatory restrictions on dividends, the Company’s ability to pay dividends depends to a large extent on the amount of dividends paid by the Bank and any other subsidiaries. The Bank, as a North Carolina banking corporation, may pay dividends only out of undivided profits as determined pursuant to North Carolina General Statutes Section 53-87. As of December 31, 2003, the Bank had undivided profits of approximately $96.6 million. Additionally, current federal regulations require that the Bank maintain a ratio of total capital to assets, as defined by regulatory authorities, in excess of 6%. As of December 31, 2003, this ratio was 14.33% for the Bank, leaving approximately $77.4 million of the Bank’s undivided profits available for the payment of dividends.

5

In an effort to achieve a measurement of capital adequacy that is more sensitive to the individual risk profiles of financial institutions, the various financial institution regulators mandate minimum capital regulations and guidelines that categorize various components of capital and types of assets and measure capital adequacy in relation to a particular institution’s relative levels of those capital components and the level of risk associated with various types of assets of that financial institution. The FDIC and the FRB statements of policy on “risk-based capital” require the Company to maintain a level of capital commensurate with the risk profile assigned to its assets in accordance with the policy statements. The capital standards call for minimum total capital of 8% of risk-adjusted assets. At December 31, 2003, the Company’s tier 1 ratio and total capital ratio to risk-adjusted assets were 17.1% and 18.4%, respectively. The Company’s leverage ratio at December 31, 2003 was 14.1%. The Company is in compliance with all regulatory capital requirements.

The Bank is subject to supervision and regulation, of which regular bank examinations are a part, by the FDIC and the North Carolina State Banking Commission (the “Banking Commission”). The Bank is a member of the FDIC, which currently insures the deposits of each member bank to a maximum of $100,000 per depositor. For this protection, each bank pays a semi-annual statutory assessment and is subject to the rules and regulations of the FDIC.

Federal banking laws applicable to all depository financial institutions, among other things, (i) afford federal bank regulatory agencies with powers to prevent unsafe and unsound banking practices; (ii) restrict preferential loans by banks to “insiders” of banks; (iii) require banks to keep information on loans to major shareholders and executive officers, and (iv) bar certain director and officer interlocks between financial institutions. The prohibitions against preferential loans and certain director and officer interlocks may inhibit the ability of the Bank and the Company to obtain experienced and capable officers and directors, to replace presently proposed officers and directors, or to add to their number.

The Company is an “affiliate” of the Bank within the meaning of the Federal Reserve Act, which imposes restrictions on loans by the Bank to the Company and on investments by the Bank in the stock or securities of the Company, which serve as security for loans by the Bank to any borrower. The Company is also subject to certain restrictions with respect to engaging in the business of issuing, underwriting and distributing securities.

Shareholders of banks (including bank holding companies which own stock in banks) may be compelled by bank regulatory authorities to invest additional capital in the event their banks experience either significant loan losses or rapid growth of loans or deposits. In addition, the Company may also be required to provide additional capital to any additional banks which it acquires as a condition to obtaining the approvals and consents of regulatory authorities in connection with such acquisitions.

Granite Mortgage, as a mortgage bank, is regulated by the Banking Commission. Because Granite Mortgage is a nonbank subsidiary of a bank holding company, it is also regulated by the FRB. In addition, because Granite Mortgage underwrites mortgages guaranteed by the government, it is subject to other audits and examinations as required by the government agencies or the investors who purchase the mortgages.

The Company cannot predict what other legislation might be enacted or what other regulation might be adopted or, if enacted or adopted, the effect thereof.

EFFECTS OF GOVERNMENTAL MONETARY POLICY AND ECONOMIC CONTROLS

The Company is directly affected by governmental monetary policy and by regulatory measures affecting the banking industry in general. Of primary importance is the FRB, whose actions directly affect the money supply and, in general, affect banks’ lending abilities by increasing or decreasing the cost and availability of bank credit in order to combat recession and curb inflationary pressures in the economy by open market operations in the United States government securities, changes in the discount rate on member bank borrowings, and changes in reserve requirements against bank deposits.

6

Deregulation of interest rates paid by banks on deposits and the types of deposits that may be offered by banks have eliminated minimum balance requirements and rate ceilings on various types of time deposit accounts. The effect of these specific actions and, in general, the deregulation of deposit interest rates have increased banks’ costs of funds and made them more sensitive to fluctuations in money market rates. In view of changing conditions in the national economy and money markets, as well as the effect of actions by monetary and fiscal authorities, no prediction can be made as to possible future changes in interest rates, deposit levels, loan demand or the business and earnings of the Company.

INVESTMENT POLICIES

For a discussion of the Company’s investment policies, see “Investment Securities” on page 31 of Management’s Discussion and Analysis of Financial Condition and Results of Operations section of this annual report.

LOAN PORTFOLIO

For a discussion of the Company’s loan portfolio, see “Loans” and “Provisions and Allowances for Loan Losses” beginning on page 27 of Management’s Discussion and Analysis of Financial Condition and Results of Operations section of this annual report.

AVAILABLE INFORMATION

Additional information about the Company and its business is available at the Company’s website, at www.bankofgranite.com. The Company’s filings with the Securities and Exchange Commission, including annual reports on Form 10-K, quarterly reports on Form 10-Q, current reports on Form 8-K, and amendments to those reports files or furnished pursuant to Section 13(a) or 15(d) of the Exchange Act of 1934, are available, free of charge, on the Company’s website at www.bankofgranite.com under the heading “Investor Relations — SEC Filings.” These reports are available as soon as reasonably practicable after the Company electronically files such material with, or furnishes it to, the Securities and Exchange Commission. Information included on the Company’s website is not incorporated by reference into this annual report.

ITEM 2 - PROPERTIES

The Bank both owns and leases its facilities as indicated in the table below. The Bank’s management considers its facilities well maintained and sufficiently suitable for present operations.

7

8

Granite Mortgage leases all of its facilities which are listed below. Granite Mortgage’s management considers its facilities well maintained and sufficiently suitable for present operations.

9

ITEM 3 - LEGAL PROCEEDINGS

There were no significant legal proceedings as of December 31, 2003.

ITEM 4 - SUBMISSION OF MATTERS TO A VOTE OF SECURITY HOLDERS

There were no matters submitted to a vote of shareholders in the fourth quarter of 2003.

10

PART II

ITEM 5 - MARKET FOR THE REGISTRANT’S COMMON EQUITY AND RELATED SHAREHOLDER MATTERS

The Company’s common stock, $1 par value, trades on The NASDAQ National Market^® tier of The NASDAQ Stock Market^® under the symbol GRAN. Price and volume information is contained in The Wall Street Journal^® and most major daily newspapers in the NASDAQ section under the National Market System listings.

During 2003, the market participants making a market in the Company’s common stock with the highest volumes of Company shares traded were Goldman Sachs & Company, the National Stock Exchange, Knight Equity Markets, LP and Wachovia Capital Markets.

As of December 31, 2003, there were 13,600,182 shares outstanding, owned by approximately 2,700 shareholders of record and an estimated 4,000 holders of shares registered in street name or as beneficial owners. The following table presents the quarterly market sales prices and dividend information for the two years in the period ended December 31, 2003.

Quarterly Common Stock Market Price Ranges and Dividends

* Amounts for periods prior to May 31, 2002 have been restated to reflect the 5-for-4 stock split paid May 31, 2002.

The following table sets forth information as of December 31, 2003 regarding shares of the Company’s common stock that may be issued upon exercise of options previously granted and currently outstanding under the Company’s stock option plans, as well as the number of shares available for the grant of options that had not been granted as of that date.

11

ITEM 6 - SELECTED FINANCIAL DATA

The following table presents selected consolidated historical financial data of the Company for the periods indicated. This financial data should be read in conjunction with the Company’s consolidated financial statements and notes thereto.