Data Processing: The Evolution of Storage, Databases, and Transmission
Electronic data processing, also known as business information processing, involves using automated methods to handle commercial data. This typically entails performing simple, repetitive tasks on large volumes of similar information. Common examples include updating inventory stocks, processing banking transactions for customer master files, managing airline reservation systems, and utility billing.
Around 1960, the terms "electronic" or "automatic" were frequently added to "data processing" (DP) to distinguish computer-driven tasks from traditional human clerical work.
Key Facts
- The Ferranti Mark 1 was the first commercially available general-purpose electronic computer.
- Digital storage capacity first exceeded analog storage capacity in 2002.
- By 2007, approximately 94% of worldwide data was stored digitally.
- The first hard disk drive was introduced by IBM in 1956 as part of the 305 RAMAC system.
- Relational database management systems (RDBMS) became commercially available in 1981 via Oracle.
The History of Data Storage
Early computing relied on punched tape—long strips of paper with holes representing data—a technology used by machines like Colossus that is now obsolete.
Modern electronic storage began during World War II with the development of delay-line memory, specifically the mercury delay line, which was used to clear radar signals. The Williams tube, based on a standard cathode ray tube, served as the first random-access digital storage device. However, both delay-line memory and the Williams tube were volatile, meaning they required continuous refreshing and lost all data once power was removed.
The first non-volatile storage (data that persists without power) was the magnetic drum, invented in 1932 and utilized in the Ferranti Mark 1.
Storage evolved rapidly with IBM's 1956 introduction of the hard disk drive. While analog devices dominated until 2002, digital storage has since become the standard. By 2007, digital storage accounted for 94% of all data, distributed across hard disks (52%), optical devices (28%), and digital magnetic tape (11%). Global electronic storage capacity grew from under 3 exabytes in 1986 to 295 exabytes in 2007, doubling roughly every three years.
Database Management Systems
Database Management Systems (DMS) emerged in the 1960s to enable the fast and accurate retrieval of massive datasets. One of the earliest was IBM's Information Management System (IMS), a hierarchical storage system still in use today. In the 1970s, Ted Codd proposed the relational storage model, which utilized set theory, predicate logic, and the structure of tables, rows, and columns. This led to the 1981 release of the first commercial RDBMS by Oracle.
A defining characteristic of all DMS is the database schema, where the structure of the data is defined and stored separately from the data itself. This architecture allows multiple users to access data simultaneously while maintaining its integrity.
In the late 2000s, the Extensible Markup Language (XML) became a prominent format for data representation. As an evolution of Standard Generalized Markup Language (SGML), XML is both machine- and human-readable. While it can exist in standard file systems, it is often stored in relational databases for better stability and verification.
Data Transmission and Manipulation
Data transmission consists of three phases: transmission, propagation, and reception. It is categorized into two main types: broadcasting (unidirectional downstream) and telecommunications (bidirectional upstream and downstream).
Since the early 2000s, XML has been used for data interchange, particularly in web-oriented protocols like SOAP, focusing on "data-in-transit" rather than "data-at-rest."
The Pace of Technological Growth
Research by Hilbert and Lopez indicates an exponential increase in computing capacity between 1986 and 2007. The doubling rates for per capita capacity were as follows:
- Application-specific computing: Every 14 months.
- General-purpose computing: Every 18 months.
- Telecommunication capacity: Every 34 months.
- Storage capacity: Every 40 months (approx. 3 years).
- Broadcast information: Every 12.3 years.
Data Mining
To prevent archives from becoming "data tombs"—archives that are rarely visited—the field of data mining emerged in the late 1980s. Data mining is the process of discovering knowledge and interesting patterns within large volumes of data.
| Technology/Metric | Key Milestone/Detail | Era/Date |
|---|---|---|
| Magnetic Drum | First non-volatile storage | 1932 |
| Hard Disk Drive | Introduced by IBM (305 RAMAC) | 1956 |
| IMS (IBM) | Early hierarchical DMS | 1960s |
| RDBMS (Oracle) | First commercial relational database | 1981 |
| Digital vs Analog | Digital capacity exceeds analog | 2002 |
Frequently Asked Questions
What is the difference between volatile and non-volatile storage?
Volatile storage, such as the Williams tube or delay-line memory, requires a continuous power supply to maintain data and loses information when powered off. Non-volatile storage, such as the magnetic drum, retains data even after power is removed.
How does a relational database differ from a hierarchical one?
A hierarchical database, like IBM's IMS, stores data in a tree-like structure. A relational database, proposed by Ted Codd, organizes data into tables with rows and columns based on set theory and predicate logic.
What is a database schema?
A database schema is the structural definition of the data contained within a database, which is stored separately from the actual data itself.
What is the purpose of data mining?
Data mining is used to discover interesting patterns and knowledge from massive amounts of data, preventing these datasets from becoming "data tombs" that are stored but never analyzed.
What is XML and why is it useful?
The Extensible Markup Language (XML) is a text-based data representation format that is both human- and machine-readable. It is widely used for data interchange and web-oriented protocols like SOAP.