Quickbase platform performance, 2021 and beyond

By Harrison Hersch posted 03-30-2021 11:57

  

March 3rd, 2021 was the busiest day in terms of traffic in Quickbase’s 20-year history. We now regularly exceed two billion requests (a request is simply any human user or API traffic) in a given month all while at the same time, builders are creating applications with a higher complexity than ever before. Keeping the platform, and individual applications, humming requires an awesome Site Reliability Team whose primary purpose is to keep our platform up and running while balancing security and compliance.

On top of the platform, and without a single line of code, our innovative builders have created applications that exceed our wildest dreams. Some apps have 270+ tables. Some have 750+ roles and others have 10,000+ reports. And late 2020, an individual table managed to get to 50,000 fields. To those of you that are thinking it – this is not a challenge. We aren’t encouraging you to try and beat those records but rather highlight the ever-growing complexity and scale we love to tackle.

And if nothing else, 2020 taught us to be ready for anything. With customers using Quickbase for increasingly mission-critical use cases like vaccine distribution, we need to be right there with them.

Our journey ensures that use cases targeting operational data can continue to scale on Quickbase and drive more complex insights faster. And faster means both how long it takes to create the solution, and how fast our purpose-built, in-memory, C++ database can actually “crunch the numbers”.

As it relates to core performance under-the-hood, we categorize our work into three main categories:

  1. Infrastructure updates
  2. Software optimizations
  3. Helping customers maintain their applications

Let’s dive into each one these and also discuss some exciting new developments.

Infrastructure

Our hybrid infrastructure is state of the art. At the edge is Cloudflare, a recognized world leader in providing security for web applications. In December 2020 alone, Cloudflare blocked about 2,000,000 malicious requests – keeping your data and applications safe. With an average response time of 65ms and peaking at thousands of requests per second, our infrastructure is the beating heart of the platform.

In the last 18 months, we have increased our compute capacity by over 180%. Just this month, we installed new hardware in our collocated, redundant, datacenters. And coming up in the next few months, we will be making additional hardware and operation system upgrades in an effort to achieve boosts in performance for certain areas of the platform.

This is all while we invest significantly in each layer of the stack to improve performance, reliability, and security.

Software Optimizations

If our infrastructure is the heart of the Quickbase platform, our software is the brain. Because we built, own and maintain the special sauce (in-memory database), we can optimize and tweak to an extent not readily available on other platforms. This directly benefits our customers. We constantly fine-tune the engine for use cases that Quickbase is designed for. Those are things like optimizing for speed and complexity, not size. They range from updates to Visual Studio and low-level frameworks, tweaking the bits and bytes of memory storage and deciding when to cache vs when to recalculate on-the-fly. These are all things we take on, so you don’t have to.

Working with many of you, we have evaluated application build patterns to make regular improvements to Quickbase. This partnership benefits every single application. Month over month, we act on our findings and ship incremental improvements. Without code commits, merge conflicts, or compiling a build, all applications running on Quickbase will instantly wake up faster.

What becomes a little more challenging is making software optimizations that may benefit one application pattern but degrade another. An example is the age-old wisdom that holds true in basically any database - “filter the most records first in your query”. This is true in most cases, but not all! In fact, a user in Quickbase might have an extremely complex formula or summary field which is very expensive to calculate, but does filter out the most records. We’ve seen a single field sprawl out to millions of data points in milliseconds. If this is the case, it may actually have the opposite impact to performance by performing the costliest filter on more records. With new application patterns being created all the time, we have to invest heavily in testing a dizzying array of possibilities. This leads to the tight partnership my colleagues and I in R&D have with our customer-facing teams. We work together to prioritize these improvements, minimize risk, and communicate with customers.

Out of that partnership was born a brand-new summary function, called “Has any children”. One of the most common patterns in Quickbase permissions is when view or modify permissions on a parent are based on the children. For example, “Only allow users to view projects when they are assigned to a child task”. In this case, Quickbase needs to count all of the children records on the fly. While we do this wicked fast, we found an opportunity to be even faster. For use cases where the “number of tasks” (or any other child record) is irrelevant, and just knowing if “any exist”, these new types of summary fields will calculate much faster, compared to counting all child records. Builders can begin taking advantage of this pattern in our April release.

Helping customers maintain their applications

Quickbase enables you to build applications far more quickly than you could using traditional technologies, while also requiring less maintenance. Builders don’t need to worry about updating software libraries, patching operating systems or any of the other minutia that normally accompanies software.

In the last 12 months, we have added functionality and solutions to help offer more visibility into applications. You can now gather information about field usage and reports from our RESTful APIs, which many of you use to document and manage applications. In fact, Ryan Pflederer used these APIs to lead the top rated session at Empower 2020: Automate your App Documentation and Management. And with Platform Analytics on the horizon, account and realm administrators will have more visibility than ever into their Quickbase ecosystem.

To assist builders with keeping their applications humming along, and building scalable architectures, our team of solution architects and customer success teams regularly work with you all to create centers of excellence and diagnose, advise, and architect enterprise-grade solutions. We’ve also invested in tooling for internal resources. Our teams can now scan an application’s schema to determine its memory footprint and assess impact of different elements like fields and formulas. They then use this to tailor recommendations unique for each customer and drive thought leadership - like how to best optimize user functions for speed.

I’m also happy to start talking about a new tool we are piloting in an early closed beta, an intelligent performance optimizer. This performance optimizer is a set of tools that dissect an application looking for different things it can tune. Not only are we implementing the lessons learned from 20 years and millions of applications being run on Quickbase, but this also allows us to address the challenge I mentioned above, where we don’t want to change how Quickbase behaves in a way that may benefit one customer – but degrade performance for another. The things we look to optimize can loosely be separated into two categories:

  1. Build patterns: things like the filter ordering, how permissions are set up, how a formula is written, and much more.
  2. Specific software optimizations: these are low-level tweaks that are nearly impossible to broadly roll out but now can be applied app-by-app, or even table-by-table, based on the unique characteristics of the schema.

In some of our early tests, we are seeing improvements to reports and queries ranging from 10%, all the way to a 300% – and we are just getting started.

I’m very proud about the progress our teams have been making and will continue to make over the coming months and years. So, what can you do right now?

First, one of the most impactful things that can tune the Quickbase engine is optimizing searchable fields. Next, make sure you are familiar with our performance optimizer and develop a routine where you are regularly profiling the most common actions in your application. Pay special attention to test as role/user if the app has custom permissions. You’ll want to test as different roles or users to see how things perform for them. If you’d like help tweaking certain actions, our Care team is happy to help.

Finally, reach out to your account team if you’d like to have a more global review of your Quickbase ecosystem and discuss how we can help further.

I’ll be sharing more updates as we begin rolling out these performance improvements so stay tuned.

Permalink