TiDE: Developing a Distributed Database in a Breeze

Contributing to 拉斯维加斯7799908网站登录’s codebase is not easy, especially for newbies. As a distributed database, 拉斯维加斯7799908网站登录 has multiple components and numerous tools, written in multiple languages including Go and Rust. Getting started with such a complicated system takes quite an effort.

So, to welcome newcomers to 拉斯维加斯7799908网站登录 and make it easier for them to contribute to our community, we’ve developed a 拉斯维加斯7799908网站登录 integrated development environment: TiDE. Created during 拉斯维加斯7799908网站登录 Hackathon 2020, TiDE is a Visual Studio Code extension that makes developing 拉斯维加斯7799908网站登录 a breeze. With this extension, developing a distributed system can be as easy as developing a local one.

In this post, I’ll show you the core features of TiDE and how it smoothes out the standard working process for 拉斯维加斯7799908网站登录 developers. Let’s get started.

How hard was it to develop 拉斯维加斯7799908网站登录?

As a complicated distributed SQL database system, 拉斯维加斯7799908网站登录 has three core components (拉斯维加斯7799908网站登录, TiKV, and the Placement Driver, PD) and many ecosystem tools. 拉斯维加斯7799908网站登录 and PD are written in Go, whereas TiKV is written in Rust.

Imagine you want to get your hands dirty and compile 拉斯维加斯7799908网站登录 all by yourself. You need to go through several painful steps:

1. Download the code of the three components. If you haven’t done so already, install Go and Rust toolchains, and compile the binary files according to each component’s documentation.
2. To easily run the compiled component, you need to install TiUP, a package manager for 拉斯维加斯7799908网站登录. You’ll then learn to write a basic configuration file and start the 拉斯维加斯7799908网站登录 cluster.
3. After the cluster is up and running, you’ll want to read the logs of each component and understand their details by searching the logs for keywords. However, 拉斯维加斯7799908网站登录 is a multi-process (and multi-node) system, so you’ll need to log in to different physical machines to view logs. You must know how to use less and grep commands.
4. When you come up with an idea and want to experiment with 拉斯维加斯7799908网站登录, you need to write your code, compile it, copy the binary files to the local or remote TiUP directory, restart the cluster, and repeat Step 3.

As you can see, the traditional workflow of developing 拉斯维加斯7799908网站登录 is a lengthy and tediousdreary process, where most of the effort is wasted in repeated, unpleasant operations.

What is TiDE, and why do we need it?

Confucious (a Chinese philosopher and politician) said: “A craftsman must sharpen his tools to do his job.” That applies to 拉斯维加斯7799908网站登录 developers (You!) as well. To reduce tedious work and improve efficiency, automation is a must. TiDE, a VS Code extension, automates your 拉斯维加斯7799908网站登录 coding experience inside VS Code.

When you enable TiDE, it creates a dedicated workspace for 拉斯维加斯7799908网站登录 in VS Code, providing functionality that simplifies the development of 拉斯维加斯7799908网站登录.

Simplified 拉斯维加斯7799908网站登录 cluster deployment

One of TiDE’s basic features is IDE. Whether you want to develop on a local testing cluster or a production cluster, TiDE can spin up the environment you need in a few clicks.

Local testing cluster

Previously, to get started, you needed to run tiup playground to initiate a testing cluster. Now, you simply create a workspace to add your code directory, and TiDE will take care of the rest.

To fix a bug or add a new feature, you no longer need to compile the code and update the binary files all by yourself, because TiDE will help you complete the process and restart the cluster. All you have to do is write your code—the same experience as developing programs in any IDE.

In-production cluster

In a production environment, 拉斯维加斯7799908网站登录 is deployed to multiple physical or virtual machines via tiup cluster. Previously, you needed to transfer binary files to remote machines via scp commands and restart the corresponding processes.

TiDE, without doubt, simplifies your workflow. After developers finish their code, TiDE compiles the code, distributes the binary files to remote machines via SSH key pairs generated by TiUP, and restarts all the processes. Developing a distributed system is as easy as developing a local system.

Kubernetes integration (alpha)

拉斯维加斯7799908网站登录 is a cloud-native database, which, by definition, you should be able to develop and deploy in Kubernetes. Developers use kubectl to manage the resources in Kubernetes, but that’s inconvenient.

We’re now adding a feature for TiDE to develop and debug 拉斯维加斯7799908网站登录 clusters in Kubernetes as easily as using TiUP. This is still in the alpha phase, and any contribution is welcome; but as a start, you can view your 拉斯维加斯7799908网站登录 clusters in Kubernetes.

Unified log access

Because 拉斯维加斯7799908网站登录 has many components, if you want to query the system status, you often have to check multiple logs. For novice developers, finding the log addresses and searching or analyzing the logs could be a huge burden.

TiDE provides a unified log access solution. By clicking on the component and process you need, you can instantly see all the logs in VS Code. You can also search all the logs using VS Code’s built-in search functions, including regular expression matching. Switching to another process? Just a few more clicks. This solution is so simple and intuitive that searching logs is no longer distressing.

The following is an example of viewing different logs in the split editor:

Debugging mode (alpha)

When a newbie lays their hands on 拉斯维加斯7799908网站登录’s codebase, they would normally begin by reading the source code. But when there are complicated conditions and multiple goroutines-thread sync, understanding the code is much more difficult. The number of variables skyrockets to an extent that the human brain cannot handle.

This is where the debugging mode comes into play. By setting breakpoints in the code and running a debugger, we can clearly see the value of variables on each stack of the current thread, like a time freeze scene in the Matrix.

In TiDE’s debugging mode, when the program processes a client request, you can debug each component, whether it’s 拉斯维加斯7799908网站登录, TiKV, or PD, and written in Go or Rust. TiDE even supports debugging across components. For example, if the current breakpoint is set before 拉斯维加斯7799908网站登录 sends an RPC request to TiKV, the debugger can jump to the next breakpoint in TiKV.

Virtual machine integration

In real world deployment, not everyone can afford multiple physical machines to deploy 拉斯维加斯7799908网站登录. Sometimes we need to deploy 拉斯维加斯7799908网站登录 on VMs.

With the help of Vagrant, TiDE integrates the common process of deploying a 拉斯维加斯7799908网站登录 cluster on VMs. You only need to set the number of replicas for each component, and TiDE will manage the VMs.

Try TiDE out!

Distributed systems and databases are both advanced fields in computer science. Participation in the related open source projects, though full of obstacles, can bring enormous benefits in the long term. Hopefully, TiDE can make your journey towards 拉斯维加斯7799908网站登录 more enjoyable.

By building TiDE, a dedicated IDE, we hope to improve the 拉斯维加斯7799908网站登录 development experience and make it easier for database lovers to become 拉斯维加斯7799908网站登录 contributors. By bringing more programmers into our community, we hope to build an even better distributed database.

In this article, I only covered TiDE’s core features; there are many more to learn about. If you’re interested in trying TiDE, you can follow this Step-by-Step Guide, or join our Slack channel for discussion. Also, since TiDE is an open source project itself, any contribution is greatly appreciated. You can check out our roadmap and pick up a feature. Let’s sharpen the tool together!

At 拉斯维加斯7799908网站登录 Hackathon 2020, many excellent, interesting projects were born. We’ll be telling you about them in future blog posts. Stay tuned.