Introduction

I am pleased to introduce a significant tool that we are releasing in Quickbase called Performance Insights. This will help builders of Quickbase apps diagnose and resolve performance hotspots within minutes, without getting a PhD in software architecture. I want to share why this is such a big deal and the describe the unique challenges that we face at Quickbase when it comes to performance.

What makes Quickbase special?

I joined Quickbase more than 20 years ago as an early engineer. Over the course of those twenty years, I've had the great opportunity to work on many of the core features. Even after all that time, I'm constantly amazed by the vision of the Quickbase founders. You can hear more from the founders, who I was lucky to have worked with, towards the end of this keynote presentation. Their vision was both bold and simple – let anyone (not just computer scientists) build applications of any level of complexity without writing code. The software will do all the heavy lifting, hide all the complexity, and enable the business user to create and iterate on their app within minutes – solving their unique business problems. No hardware to provision, no schema to design, no code to write, and no security threats to plan around. Finally, and most pertinent to this post, your app grows as your needs grow (read more about apps scaling with your business on the Quickbase blog).

 
I want to underscore that we were doing all of the above before the industry invented terms to describe many of these technologies. For example, we were doing multi-tenancy, no code / low code, SaaS, cloud hosting, and in-memory database. We certainly did not invent all of these, but it's been a real thrill to be able to say that some of us worked on a product that had characteristics that needed the software industry to invent categories to describe them.

Under the covers

So, how do we help with the heavy lifting when it comes to building apps that solve your unique business problems? One of the key decisions a database engineer makes is whether to use a transactional or an analytics database. The reason that is an important decision is that the usage patterns and therefore the design philosophy of the two are very different. Transactional databases are optimized for transactions – creating records, updating records, deleting records and retrieving limited records for well-understood queries. Analytics databases, on the other hand, are used to analyze data – to slice, dice, and drill into data. Queries are not well understood in advance, but they evolve as the user finds interesting patterns. Optimal performance around transactions is usually not a key requirement.

 

Quickbase apps are used for both purposes. Builders not only use their apps to get data in and out of their apps and run workflows, but they also use our reports and dashboards to analyze the data. Delivering a no-code software platform that delivers the best of both worlds required us to innovate on the core technology. We therefore wrote our in-memory database engine that optimized for these hybrid use cases. Our engine specifically is written to handle rapid schema changes, optimizing transactions, and queries dynamically to adapt to those schema changes and hiding complexity.

What is Performance?

At this point, you might be wondering why we need Performance Insights. Haven't I told you about all the wonderful things our core engine does and how it does the heavy lifting for you? Yes it does, but only up to a point. The overwhelming number of our builders (over 90%) will never use the Performance Insights feature because they will never have to consider scale. However, there is a limit to the magic of any software and at some point certain design choices do impact performance. This is no different than having a database adminstrator tune an Oracle or MySQL database for indexes, storage, memory, etc. It is just that the dimensions that we ask you to consider to tune your app are much fewer, and the threshold at which those become necessary for the types of apps we support is much higher than those database products. Using a real-world analogy, a Tesla is much easier to maintain than a conventional car, but it still needs its tires rotated.

 
There is a great blog post written by one of my colleagues at Quickbase that explains in detail some of the dimensions around the performance of a Quickbase app, so I won't describe those here. However, I do want to explain what experienced Quickbase solution architects consider when they review a customer's app to improve its performance, which will hopefully illuminate the value of the Performance Insights feature.

 

When an app is experiencing performance issues, it is often related to some request that is taking a long time. For example, queries might be taking too long to run, or adding/deleting records might be slow. However, the root cause behind those can often be challenging to unearth. The reason behind that is our desire to shield customers from complexity. We define complexity as the myriad of permutations and combinations that go into creating a valuable app. Those include but are not limited to tables, fields, relationships, formulas, lookups, summaries, reports, permissions, etc. Often, the interaction of all or some of these features can result in performance issues. For example, when a builder creates a summary field, they are implicitly creating an aggregate query on a table. If that summary field is conditional, that is an additional query clause. If that summary field is summarizing a formula field, it is aggregating a calculated field on-the-fly. You can see how the complexity multiplies and we haven't even discussed this summary field referring to other summary or lookup fields that span multiple tables! Now consider other attributes of a Quickbase app like lookups, permissions, formula queries etc. All of these are common to Quickbase but would require very complex queries or application logic in normal databases. Exacerbating all of these is the fact that no two apps in Quickbase are quite the same. This makes sense, as the power of Quickbase is the ability to build customized apps that are right for you. However, with that variety comes the very challenge that each app's performance characteristics are slightly unique.

 

So far, I have explained why it is challenging to unearth performance issues for a given app. That leaves the question of how we discover the root cause of the issues and help our customers improve their app's performance. The best answer to that is that we apply our 20+ years of experience in debugging performance issues. For example, we use our internal logs, our heuristics on which parts of the schema and data might potentially be affecting performance, reviewing disk I/O times and sometimes even ask customers' permission to access their apps so that we can step through their queries in a granular way by attaching our code debuggers.

 

Just one year ago, we released Performance Optimizer - a tool that scans your entire application and makes changes, safely, in production, to dramatically improve performance. This capability has already improved hundreds of applications. Taking this to the next level usually requires experienced solution architects and engineers to facilitate investigations. However, we aspire to do better and in keeping with our philosophy, we want to give builders the power to debug their own apps. So, the question is – how do we enable you to debug your apps? Enter Performance Insights.

Performance Insights?

One of the key motivations behind Performance Insights is that we realized that even though our solution architects drew on their vast experiences to debug highly customized apps, there was a set of heuristics that was common to the act of debugging apps. In other words, no matter the app, they looked at certain "tells" that either had to do with the way the schema was designed, the nature of the record data, the way queries were structured, the way formulas were written, or access patterns across the app (accidentally looping API calls). If we think of these heuristics as a decision tree where each node represents a set of contributing factors towards sub-optimal performance, the task becomes ruling out these possibilities to reach the root cause. With the Performance Insights feature, we've been able to automate these rules. We have taken the collective learnings of our most experienced engineers and architects and coded that decision tree so that you – the builder - can use these outcomes to improve the performance of your app.

 

So, let's look at Performance Insights in action.

 

When you tell us to run Performance Insights, we do a few things:

 

1. Begin logging all requests and associated meta-data.
2. Run a series of checks against the app to gather information on how it is built.

 

When the insights window elapses, we:

 

1. Aggregate all the logs, which can be millions of data points.
2. Overlay that with the app heuristics we understood.
3. Compare these combinations with patterns and anti-patterns we know about from our two decades of app-building knowledge.

 

In the example below, let's look at an inefficient import. First, gathering this information would be difficult in a traditional stack, since many things are in separate layers.

 

Whereas many other solutions, even claiming to be no-code, focus on providing log visibility and telling you that the import operation spiked your CPU usage, we aspire to do better. In this case, we have analyzed the import transaction time, cross-referencing the uploaded dataset with all the records in the destination table to determine how efficient the import was. Then, using what we know about compute effectiveness and import design, we surface an actionable insight to you.

Over time, we will be enhancing this engine to add more heuristics to capture even more learnings.

Conclusion

Quickbase is an amazing platform to create applications rapidly and iterate as your needs evolve. It is designed to empower builders to build powerful apps without worrying about a myriad of issues that traditional software requires one to plan for. It does this by shouldering most of the complexity on scale, performance, security, deployment, and hosting. Historically, when the small number of apps required performance tuning, we asked our experienced engineers to help diagnose the issues and recommend changes in app design and access patterns to improve performance. With the release of Performance Insights, we are putting this debugging power in the hands of our builders so that they can diagnose performance issues in near real time and mitigate them. We still anticipate that periodically there may be certain apps that will require our engineers to diagnose the root cause and help with the mitigation. Even in those cases, it is our intention to use those learnings to enhance the power of Performance Insights and provide even more insights into how your apps are performing. If you are interested in knowing more about Quickbase and performance, check out this excellent Empower talk on performance and the many considerations that go into it.

------------------------------
Soumya De
Chief Architect, Quickbase
https://www.linkedin.com/in/soumyade1/
------------------------------