Water created the 4,000+ caves in Virginia by dissolving limestone. Rainwater seeped down through the soil, becoming slightly acidic as it passed through decaying organic matter such as leaves. The acidic water slowly transformed the calcium carbonate - CaCO4, the predominant mineral in limestone and dolomite. (Dolomite includes more magnesium than ordinary limestone, and is formed when magnesium ions replace calcium ions in the original limestone formation.)
The chemical weathering created calcium bicarbonate - Ca(HCO3)2, which easily dissolved in water. Biological activity by sulfur-based microbial communities may also contribute, as at Cesspool Cave along Sweet Springs Creek in Allegheny County.1
As the calcium carbonate weathered away, the rock became pockmarked with voids and created a "karst" landscape with springs, sinkholes, caves. In Virginia, eroded remnants of cave systems are visible at Natural Bridge, Natural Tunnel, and Natural Chimneys. Acid rain may also be speeding up the creation of caves, by increasing the acidity of rainwater.
The chemical weathering process is invisible when it occurs underground, but we can see the same process in cemeteries aboveground. On old marble gravestones, once-clear letters have eroded away. The calcium in the marble (which is metamorphosed limestone) has dissolved in just a few decades, until the carved letters on many gravestones are no longer legible.
Some minerals, such as silicon dioxide or quartz (SO2), are very hard to dissolve. Go to a Virginia beach and you'll see predominantly quartz sand grains, rather than calcium carbonate grains. Quartz is not very reactive; silicon dioxide is about the last mineral to dissolve, as rocks are washed down from the Appalachians. The quartz resists its inevitable fate of dissolving into the ocean. Granite headstones, with a high percentage of quartz and little or no limestone, retain their lettering longer than marble headstones.
The granite rocks of the Blue Ridge, and the sandstone ridges of Massanutten Mountain, are not riddled with caves like the limestone valleys in Virginia. The metamorphic bedrock of the Piedmont also lacks caves. There are only a few places in the Piedmont where limestone outcrops on the surface and caves might form naturally.
Where calcium carbonate (limestone) is the bedrock, caves will be more common. A map of cave locations in Virginia shows that nearly all the caves are west of the Blue Ridge, in the limestone Shenandoah Valley and the equivalent valleys south of Augusta County.
Several caves are located in Loudoun County in a limestone conglomerate formation (the Leesburg member of the Balls Bluff siltstone, also known as "Calico marble") that developed during the Triassic Period. Over 1,200 feet of passage have been explored in Rust Cave #1. Sinkholes are common at Temple Hall Farm, and a window into an underground chamber is on the east side of Route 15 just north of the entrance into the Raspberry Falls subdivision.2
There is one cave in York County, where Cornwallis supposedly took shelter during the bombardment before surrendering on October 9, 1781 - but most if not all of that hole in the hillside was excavated by people during the Revolutional War battle.
The longest cave system in Virginia is the Omega System in Wise County, with nearly 30 miles of underground passages. There are over 80 cave systems in Virginia with more than one mile of passage, and at least eight caves in Virginia are at least 500' deep:3
Natural Chimneys shows how water erodes limestone
The Chimneys (including the Great Tower) were a tourist attraction even prior to the Civil War
Source: Harper's New Monthly Magazine, Virginia Illustrated (February 1855)
1. Annette Summers Engel, Megan L. Porter, Brian K. Kinkle, Thomas C. Kane, "Ecological Assessment and Geological Significance of Microbial Communities from Cesspool Cave, Virginia," Geomicrobiology Journal, Vol. 18 Issue 3 (2001), http://www.tandfonline.com/doi/abs/10.1080/01490450152467787 (last checked June 16, 2012)
2. "Minutes of the Fall 2000 VAR Meeting," Virginia Area Region (VAR) of the NSS, September 24, 2000, http://www.varegion.org/var/theVar/varMeetMinutes/minutesFall2000.shtml (last checked June 11, 2013)
3. "USA Longest Caves by State," compiled by Bob Gulden, June 17, 2014, http://www.caverbob.com/state.htm (last checked July 14, 2014)