Forum Discussion

hhersch's avatar
Qrew Assistant Captain
6 years ago

How we built Quickbase to scale and you can too!

Quickbase is an immensely powerful platform. At the core, we have built a custom database that combines the traditional database and application layers. Quickbase App Builders do not need to be database administrators or have coding backgrounds and we abstract out unnecessarily-complicated elements such as indexing. This leads to quicker time-to-value and the ability to build enterprise-grade ecosystems in a fraction of the time you might expect. However, no platform is perfect, and they all have tradeoffs.

In software, there are three fundamental components that can be optimized for, and a fourth that impacts scale:

  1. Size: How big the database and application will get. Think number of records, tables and fields.
  2. Complexity: How much "thinking" the application does. Things like reporting, calculations, workflows, etc.
  3. Speed: How fast can the application return an answer that requires computation?
  4. Concurrency: At any given time, how many requests hit the system at the same time? This generally surfaces itself in the number of users but might also be frequency of hits from integrations.

If we look at something like Amazon S3, they clearly optimize for size and speed. But, S3 is not complex (the name contains Simple). This is an intentional design decision from Amazon as to what their goal is with the product. It is generally very difficult, if not impossible, to optimize for all three since "optimization" implies some specificity and decision making. In our case, we optimize for speed and complexity. What that means is that we want builders to be able to pose very difficult questions to Quickbase by way of formulas, permissions, lookups and summaries and have Quickbase answer them ridiculously fast. Here is a Community post with an in-depth sample.

That leaves "Size" on the table, which we do not optimize for. But let's define what that really means. Quickbase currently has a production table size limit of 500mb. That is a lot of data. Opening a 500MB CSV file in Excel might not go so well. For a comparison, let's say the average Kindle book is ~2.5mb and ~295 pages. That means a single Quickbase table can hold the equivalent of about 60,000 pages of text.

Now, how we perform at those sizes depends on the other factors of complexity and concurrency. But in general, Quickbase applications are super-fast. Remember though that the 500MB of data could translate into much more RAM, which is where we do all our calculations that is how we optimize for speed and complexity. It is perfectly acceptable for applications to have gigabytes of memory in a single table.

To give an analogy, you can buy a high-end computer today and it will probably come with a 1TB hard drive (i.e., data warehouse), but likely only 16-32GB of RAM. You can load up that hard drive with pictures, videos, and other content from the dawn of time, but how often is it used? And loading things from the hard drive takes a few seconds. Your RAM, which is much faster, keeps less information running but can work with that information orders of magnitude faster. Would you rather have Microsoft Outlook open where you can recall it in a fraction of a second, or open it fresh from your hard drive where it will take longer? The way we often define this is "operational data". Think of it as the data being used to run the business on a regular basis. In an inventory and ordering system, you obviously need to have historical ordering data to predict future needs, but you might not need every single field for every single inventory transaction for ten years in a single table - this is normally the job of a data warehouse.

None of these limits prevent Quickbase from solving organizational requirements when apps get a bit large. That is when we start to discuss scale strategies. In software and infrastructure, you can either Scale Out or Scale Up. A visual of what that looks like is below. If we think about it like traffic on a highway, Scaling Up would be increasing the speed limit, or getting a faster car. Scaling Out would be adding more lanes to the highway. Scaling Out is not only how we scale our infrastructure, but how our customers can best scale their ecosystems built on the platform. As we grow, we add servers in our datacenters vs. making each server faster. When an application is created on Quick Base, our load balancers go out and determine the best place for that application amongst our vast hardware and thousands of virtual environments. What is great about this is that you, as the customer, get the benefit of spreading your applications out over virtually infinite compute resources without lifting a finger. It all happens transparently to the builder.


Quickbase is in a unique position in this new "low code"/"no code" space. In fact, we were recently recognized by Gartner as a February 2019 Peer Insights Customers Choice for Enterprise High Productivity Application Platform as a Service (hpaPaaS). That is quite a name, but what it really means is that based on our stellar reviews, we are at the intersection of scalable enterprise applications and no-code building the only true no-code hpaPaaS identified. At one end of the spectrum, there are platforms such as Outsystems or Appian that are geared towards making the lives of a software engineer easier. Applications built on these platforms will take much longer to achieve value since they still require coding but might be able to get "bigger" with proper indexing, performance optimizations and many other highly-technical considerations that we do not want our builders thinking about. In general, Quickbase can still answer more complex questions, faster, due to our unique database. On the other end, we really like what Airtable has done. We find that they have a capable set of entry-level features wrapped up in a nice interface that is very appealing for someone trying to upgrade from a spreadsheet. However, Airtable has a limit of 50,000 records for the entire application on their Pro subscription. In the middle, is a more focused solution like Salesforce. Salesforce somewhat forces simplicity via the ways of limits - a lot of them. They have made the conscious decision to allow for more vertical growth at the expense of customization, and by having so many limits that they can constrain the use cases into a very tight box. And these limits are highly technical not suited for no-code solutions.

Quickbase applications take infinite shape and can have millions of records in a table, let alone across an entire application. That is why we know we are uniquely positioned to solve many challenges an organization will have on their operational data without builders having a computer-science background. In fact, we co-exist with these other platforms regularly and think they have a great place in the suite of tools for an enterprise.

How does this translate into making good architecture decisions during your evolution with Quickbase? If you have 1,000 employees in your company that is planning to run the entire business on Quickbase, we want to set you up for success. The ideal way to do this is by setting up logical distinctions between applications. This might mean:

  1. One billing application which is very complex due to calculations of invoicing, prorating, etc. It might only be used by Accounting, so maybe 50 people. This application probably isn't very big.
  2. One application may be used for inventory purposes which isn't terribly complex, but probably big. Inventory tracking tends to have a ledger structure which requires a million or more records. It also might have relatively high concurrency if there are frequent debits/credits.
  3. Another example could be the general Human Resources Information System (HRIS). While this app might only have moderate complexity, all 1,000 users will have access but not frequently interact with it (low concurrency).

We can see from the above that Quickbase is going to cover a wide breadth of functions in this company, which is awesome! If we built all these functions inside a single application, we would then be combining complexity, size, and concurrency. And things might work out fine, but a far more scalable approach would be to spread this out over multiple applications. This not only makes management easier because a builder can administer billing without being exposed to inventory, but it also spreads all the requirements over more Quickbase horsepower.

In these cases, it is critical to have a central data management (CDM) strategy. We provide many different tools for this, and it is important to select the right one. You can find an article written by our Best Practices group here. We encourage you to reach out to our Technical Support team, or your account team here at Quickbase if you'd like to discuss how Quickbase can scale with your organization.

Keep an eye out for sessions and content that expand on these topics at Empower 2019.