The Evolution of Blockchain Data Indexing Technology: From Nodes to AI-Powered Full-Chain Services

1. Introduction

In recent years, decentralized applications (dApp) have been flourishing in the blockchain field, evolving from early financial, gaming, and social applications to the current diverse multi-chain ecosystem. During this process, the data sources and access methods relied upon by dApp have also been continuously evolving.

In 2024, artificial intelligence ( AI ) and Web3 have become hot topics. AI systems require vast amounts of data to continuously learn and evolve, just as plants need sunlight and moisture to thrive. Without data support, even the most advanced AI algorithms struggle to demonstrate their intended intelligence and efficacy.

This article will conduct an in-depth analysis of the evolution of blockchain data indexing from the perspective of data accessibility in the industry development process. We will also compare the traditional data indexing protocol The Graph with the emerging blockchain data service protocols Chainbase and Space and Time, discussing the similarities and differences between these new protocols that combine AI technology in terms of data services and product architecture.

2. The Complexity and Simplicity of Data Indexing: From Blockchain Nodes to Full Chain Databases

2.1 Data Source: Blockchain Node

The foundation of Blockchain is a decentralized network composed of numerous Nodes, each of which stores a complete copy of the Blockchain data. However, for ordinary users, building and maintaining a Node is not an easy task; it requires not only specialized skills but also faces high hardware and bandwidth costs. At the same time, the query capabilities of ordinary Nodes are limited, making it difficult to meet the needs of developers.

To solve this problem, remote procedure call ( RPC ) node providers have emerged. These service providers are responsible for the operation and management of nodes, providing data access services to users through RPC endpoints. Although public RPC endpoints are free, they have rate limits that may affect the user experience of dApps. Private RPC endpoints, while offering better performance, still exhibit inefficiencies for complex queries and are difficult to scale across networks. Nonetheless, the standardized API interfaces provided by node providers lower the barrier for users to access on-chain data and lay the foundation for subsequent data parsing and applications.

2.2 Data Parsing: From Prototype Data to Usable Data

The raw data provided by blockchain nodes is usually encrypted and encoded, making it difficult for ordinary users and developers to use these data directly. The data parsing process converts complex prototype data into a more understandable and operable format, which is a key link in the entire data indexing process.

2.3 The Evolution of Data Indexers

With the explosion of blockchain data, the demand for data indexers has become increasingly prominent. Indexers can organize on-chain data and store it in a database for easy querying. They index blockchain data and provide SQL-like query languages such as GraphQL interfaces, making data readily available. Indexers provide developers with standardized query languages, greatly simplifying the data retrieval process.

Different types of indexers have their own characteristics:

1. Full Node Indexer: Extracts data directly from full nodes, ensuring data integrity, but requires a large amount of storage and processing resources.
2. Lightweight Indexer: Relies on full nodes to fetch specific data on demand, reducing storage requirements but possibly increasing query time.
3. Dedicated Indexer: Optimized for specific types of data or blockchain, such as NFT data or DeFi transactions.
4. Aggregated Indexer: Extracts data from multiple Blockchains and sources, including off-chain information, providing a unified query interface, suitable for multi-chain dApps.

Currently, the storage requirements for Ethereum archive nodes vary from 3TB to 13.5TB across different clients, and this data volume continues to rise as the Blockchain grows. In the face of such a large data volume, mainstream indexing protocols not only support multi-chain indexing but also customize data parsing frameworks according to the data needs of different applications.

Compared to traditional RPC endpoints, indexers have significant advantages in data indexing and query efficiency. They can efficiently index massive amounts of data, support complex queries and data filtering, and perform analysis after extraction. Some indexers also support aggregating data sources from multiple blockchains, avoiding the issue of multi-chain dApps needing to deploy multiple APIs. By operating in a distributed manner, indexers provide stronger security and performance while reducing the interruption risks that centralized RPC providers may pose.

( 2.4 Full-Chain Database: Aligning to Stream Priority

As the application demands become increasingly complex, standardized API index formats are gradually unable to meet the diverse query needs, such as cross-chain access or off-chain data mapping. In modern data pipeline architecture, the "stream-first" approach has become a solution to address the limitations of traditional batch processing, enabling real-time data processing and analysis.

Blockchain data service providers are also moving towards building blockchain data streams. Traditional indexer service providers have launched products that obtain real-time blockchain data in a data stream manner, such as The Graph's Substreams and Goldsky's Mirror. At the same time, real-time data lake services based on blockchain-generated data streams, such as Chainbase and SubSquid, have also emerged.

These services aim to address the demand for real-time analysis of blockchain transactions and provide more comprehensive querying capabilities. By redefining the challenges of on-chain data through the lens of modern data pipelines, we can envision a possible world where high-performance datasets can be tailored for any business use case.

![Reading, indexing to analysis, a brief overview of the Web3 data indexing track])https://img-cdn.gateio.im/webp-social/moments-587ce87f6dbedee4acec7d939fed6980.webp###

3. AI + Database? In-depth comparison of The Graph, Chainbase, Space and Time

( 3.1 The Graph

The Graph network provides multi-chain data indexing and query services through a decentralized node network, making it easier for developers to index Blockchain data and build decentralized applications. Its main product models include a data query execution market and a data indexing cache market, serving the product query needs of users.

Subgraphs ) are the fundamental data structure of The Graph network, defining how to extract and transform data from the Blockchain into a queryable format. Anyone can create subgraphs, and multiple applications can reuse these subgraphs, enhancing data reusability and efficiency.

The Graph network consists of four key roles: Indexer, Curator, Delegator, and Developer. The Indexer provides indexing and query processing services; the Delegator stakes tokens to support the operation of index nodes; the Curator is responsible for signaling which subgraphs should be indexed by the network; the Developer creates and submits subgraphs as the demand side.

The products of The Graph are also rapidly developing in the wave of AI. Tools like AutoAgora, Allocation Optimizer, and AgentC developed by Semiotic Labs enhance the ecosystem's performance in various aspects, such as dynamic pricing mechanisms, resource optimization allocation, and natural language queries.

![Reading, indexing to analysis, brief overview of the Web3 data indexing track]###https://img-cdn.gateio.im/webp-social/moments-cf9a002b9b094fbbe3be7f611001b5c1.webp(

) 3.2 Chainbase

Chainbase is a full-chain data network that integrates all Blockchain data into one platform. Its unique features include:

• Real-time Data Lake: Provides a real-time data lake specifically for Blockchain data streams.
• Dual-chain architecture: The execution layer is built on Eigenlayer AVS, forming a parallel dual-chain architecture with the CometBFT consensus algorithm.
• Innovative data format standard: Introduce the "manuscripts" data format standard to optimize the structuring and utilization of data in the cryptocurrency industry.
• Encrypted World Model: Combining AI model technology to create an AI model that can understand, predict Blockchain transactions, and interact with them.

The AI model Theia of Chainbase is its key highlight, based on NVIDIA's DORA model, combining on-chain and off-chain data as well as spatiotemporal activities, learning and analyzing crypto patterns, and responding through causal reasoning.

3.3 Space and Time

Space and Time (SxT) is dedicated to building a verifiable computing layer that expands zero-knowledge proofs on decentralized data warehouses. Its innovative technology Proof of SQL is a type of zero-knowledge proof technology that ensures SQL queries executed on decentralized data warehouses are tamper-proof and verifiable.

SxT collaborates with Microsoft's AI Innovation Lab to develop generative AI tools that enable users to process blockchain data through natural language. In Space and Time Studio, users can input natural language queries, and the AI will automatically convert them into SQL and execute the queries.

![Read, Index to Analyze, Briefly Describe the Web3 Data Indexing Track]###https://img-cdn.gateio.im/webp-social/moments-97443cbd177ac4ffd1665da670ffbf12.webp(

Conclusion and Outlook

Blockchain data indexing technology has evolved from the initial node data sources, through the development of data parsing and indexing, to the AI-enabled full-chain data services, undergoing a gradual improvement process. The continuous evolution of these technologies has not only improved the efficiency and accuracy of data access but also provided users with an intelligent experience.

In the future, with the continuous development of new technologies such as AI technology and zero-knowledge proofs, blockchain data services will become further intelligent and secure. As infrastructure, blockchain data services will continue to provide strong support for the progress and innovation of the industry.

![Reading, indexing to analysis, brief description of the Web3 data indexing track])https://img-cdn.gateio.im/webp-social/moments-0742180b7da8a9dcddafc465a4dba9cb.webp(