Discussions

Expand all | Collapse all

Apps, Performance, and Best Practice

  • 1.  Apps, Performance, and Best Practice

    Posted 20 days ago

    I prefer to have a very limited number of apps, as management can be come quite labor intensive, and I tend to find cross app relationships with some limitations or inconveniences versus staying within a specific app. Additionally, I find a single app better for my use cases, and less complicated for users.

    I in fact of three apps, but don't have an intention to expand to more apps. One is essentially a master app, that feeds into two separate project manager apps.

    My master app has 150+ tables and growing. I am wondering if anyone has had performance issues as apps grow in size, and if best practice for best performance is to continue inside one app, or call to other apps. I would prefer to keep growing as I am now, unless convinced I will regret it down the road. Thoughts?



    ------------------------------
    Mike Tamoush
    ------------------------------


  • 2.  RE: Apps, Performance, and Best Practice

    Posted 20 days ago
    Edited by Austin K 20 days ago
    Do you use the sandbox feature? If you do then keeping your app size to under 1 GB is a huge deal. My main app is above that limit so when using sandbox mode it can bring over exactly 0 records on any tables. You must do all the imports manually yourself if you want to see "live" data to play with. So annoying to do when you have a bunch of related tables that all need something populating. You might think why not create something to import via the API and I would say yes but you cannot do an API import with the sandbox enabled(API calls the read are supported, but not ones that write data...) I have reached out to support and even talked to an employee on these forums, it doesn't seem like it is a plan to increase this 1 GB limit. Personally I am at almost 2 GB in my main app so I have a long way to go to get back below 1.

    The other annoying part about importing like we have to to get around the sandbox issue is that not all tables still contain all of their original records, some have been deleted over the years. So when you go to import a set of data and that data is related via record id to something else, it is a nightmare getting them all set up correctly again. It can be done but it is just another annoying aspect of this.

    We are busy moving multiple related tables out into their own apps to slim down our main app, so moving in the opposite direction of you. This is because we started with one massive app and built off that and now feel like it is bloated. This is not always helpful because with some ways of connecting tables it brings both apps into the same instance and you really are not saving anything on performance. I forget exactly which option does this, but it was not hard for me to find previously. If you can separate the workflow enough sometimes a new app just makes sense and that is what we are exploring now.

    I will definitely be interested to hear others experiences with this. I don't know if we are moving down the right path either but it seems to make sense for us right now.


  • 3.  RE: Apps, Performance, and Best Practice

    Posted 20 days ago
    The Performance is related conceptually by # of concurrent active uses​ times # of records in the the tables being accessed frequently times the number of summary fields on reports.  Those factors hurt performance.

    If you do go the route of separate apps, then you cannot make any cross app relationships or cross app saved table to table imports as that just makes it one giant app under the covers.  So you need to use Sync tables to replicate master files like customers or warehouses.

    If you have many hundred of users like say 500+, then user count can be a factor, probably less than that, less of a concern.  tables with hundred of thousands of records like say 200,000+ records can get to be a concern when combined with a high user counts using those big tables.  ​

    They keep incrementally improving the performance of Quick Base each year, a bit like Elon's Battery day where there was no magic bullet to announce, but Tesla but is adding up the 5%'s and 10%'s they keep finding to improve the performance.

    So  assuming no giant reports with killer summary fields and no frequently used tables approaching 500,000 records, you are probably going to be OK.    150 tables do not affect performance.  It's the load on those tables in terms of concurrent users hammering away at difficult reports with complex summary fields.

    ------------------------------
    Mark Shnier (YQC)
    Quick Base Solution Provider
    Your Quick Base Coach
    http://QuickBaseCoach.com
    mark.shnier@gmail.com
    ------------------------------



  • 4.  RE: Apps, Performance, and Best Practice

    Posted 19 days ago
    Edited by dmlaycock2000 dmlaycock2000 19 days ago

    Hi Mark,

    I'm really interested in (and potentially concerned by!) your comment:

    "If you do go the route of separate apps, then you cannot make any cross app relationships or cross app saved table to table imports as that just makes it one giant app under the covers.  So you need to use Sync tables to replicate master files like customers or warehouses."

    Could you clarify please.

    My specific situation is I have a master database (used for content management, and not the fastest due to prolific use of summary fields) and a slave which I update overnight using API_RunImport.

    The slave is used to serve read only data to our customers within our web app via API calls).

    Are you saying that these two apps are run a single thread because of the use of the import API to refresh the slave tables, and therefore I won't actually be seeing any performance gain from separating the two?

    And if this is the case, are you are saying the only way to break this and ensure separate threads for the two apps is for the refresh of the slave tables to be using Connected tables / Quicbase Sync?

    Thanks

    David



    ------------------------------
    dmlaycock2000 dmlaycock2000
    ------------------------------



  • 5.  RE: Apps, Performance, and Best Practice

    Posted 19 days ago
    Yes, unfortunately when you have a saved table to table import it does create what is called an app dependencies between the two apps exactly the same as if there was a cross app relationship. So that means that these two apps will run in a single thread. It is still possible that your users will have a better user experience with two separate apps but in terms of performance it is identical to having one giant app.
    They are used to be a suffix parameter that you could use which I will list before but I don't know if it is working today I am just on my iPad and didn't test properly so far this morning.  I was up late last night freebasing, oh I meant quickbasing,  it's about the same level of addiction.


    ?a=ListDependentDatabases
     


    ------------------------------
    Mark Shnier (YQC)
    Quick Base Solution Provider
    Your Quick Base Coach
    http://QuickBaseCoach.com
    mark.shnier@gmail.com
    ------------------------------



  • 6.  RE: Apps, Performance, and Best Practice

    Posted 19 days ago
    Lol.

    Thanks for the quick response, although not the answer I was hoping for!

    Migrating to using connected tables is going to be nightmare.

    Out of interest, when I use ?a=ListDependentDatabases it doesn't actually show the other database / tables irrespective of whether I append this to the master or slave DBID.

    Does that mean QB may have changed how these work?

    D

    ------------------------------
    dmlaycock2000 dmlaycock2000
    ------------------------------



  • 7.  RE: Apps, Performance, and Best Practice

    Posted 19 days ago
    I just did this on the dashboard of an app

    https://mycompany.quickbase.com/db/xxxxx(app dbid)?a=ListDependentDatabases

    and it listed all the dependencies.  many screens of them as this app is part of a legacy ecosystem that was build before Sync tables came out.

    ------------------------------
    Mark Shnier (YQC)
    Quick Base Solution Provider
    Your Quick Base Coach
    http://QuickBaseCoach.com
    mark.shnier@gmail.com
    ------------------------------



  • 8.  RE: Apps, Performance, and Best Practice

    Posted 20 days ago
    Regarding your concern about application management, be careful. Applications that have large numbers of tables often become very difficult to manage as well, especially if you have multiple workflows intersecting across the application. In my experience, limiting the number of applications in order to make application management easier tends to come back to bite you in the end.

    I would encourage you to really think about what workflows are being handled by your application and decide if any of these can/should live on their own. As Mark pointed out, performance is really going to be impacted by your use of cross-app relationships, cross-app imports, table sizes, and the size of data sets on these tables. Ease of management is going to be impacted by # of Workflows in a single app, # of Tables, # of Fields on each Table, # of User Roles, # of Forms/Reports on each table, # of Dashboards, etc. ​​​​​​

    You can split disparate workflows into separate applications and still have access to common tables like Customers, Resources, etc by using Sync tables. In that scenario, management may actually be easier because you've got less complexity within a smaller application, rather than running the entire business from one app and trying to figure out how to add/modify a process without causing issues with others.

    ------------------------------
    Blake Harrison - DataBlender.io
    Quick Base Solution Provider
    ------------------------------



  • 9.  RE: Apps, Performance, and Best Practice

    Posted 20 days ago
    Austin - when speaking about app size for sandbox, does that include file attachments? If so, I'm dead in the water because we track assets (equipment) and have roughly 20,000 assets, each with about 5 pictures.....or 85 gigs worth of attachments. Without attachments, my apps are tiny. 

    Blake - when you mention workflows, what exactly are you referring to? Automations etc?

    Mark - thanks for the tips. My largest table only ha maybe 20,000 records. I can see some tables growing but it will be many years I imagine until I even reach 100k in one table.

    I have a couple massive (I think? I have no frame of reference) tables of 500+ fields. So far so good, except my main table I ran out of Dynamic Form rules, so i will have to take some time to clean that up and rethink some of the fundamental design to alleviate that issue.

    I like to keep my user roles small, also for ease of management, but still need 6-8 roles. Which multiplied over three apps already starts to wear on the management. 

    Sounds like this can really have a list of pro's and cons each way.

    ------------------------------
    Mike Tamoush
    ------------------------------



  • 10.  RE: Apps, Performance, and Best Practice

    Posted 20 days ago
    Mike, on the roles issues,  note that you can have users in multiple roles.  That can help so you don't need to feel that you need extra roles for users who cross some boundaries.

    ------------------------------
    Mark Shnier (YQC)
    Quick Base Solution Provider
    Your Quick Base Coach
    http://QuickBaseCoach.com
    mark.shnier@gmail.com
    ------------------------------



  • 11.  RE: Apps, Performance, and Best Practice

    Posted 19 days ago
    I had no idea you could assign to multiple roles! Do you simply just add a second instance of the user in the app?

    ------------------------------
    Mike Tamoush
    ------------------------------



  • 12.  RE: Apps, Performance, and Best Practice

    Posted 19 days ago
    Yes, a user can be in multiple rules and you just add them to the app multiple times. Of course a user just counts as a user so you don't pay twice for the user.
    Now if the user is in multiple Roles they will get the most permissions possible based on all the Roles they were in.
    However,  If they are in multiple roles they cannot get the most dashboards possible :)  They still can only land on one dashboard and see one set of tables (in case you hide some tables by Role UI).

    When you look at the list of Roles you can drag and drop them up and down. The Roles which are highest up in the list will take priority as to which dashboard a user will land in.  ie they will land on the dashboard based on the highest Role they are in based on that drag n drop. 




    ------------------------------
    Mark Shnier (YQC)
    Quick Base Solution Provider
    Your Quick Base Coach
    http://QuickBaseCoach.com
    mark.shnier@gmail.com
    ------------------------------



  • 13.  RE: Apps, Performance, and Best Practice

    Posted 20 days ago
    Another question - I have one table that I have what seems to be an absurd number of report links (20ish). It's been a good way for my users to be able to view a lot of information in one place, but I suspect this is a bit tedious for QB? I've considered being very selective in which ones I actually display (which right now is all of them) versus just displaying a link. I suspect this helps the speed in which the record will load, but I am not sure.

    Right now, the record can be cumbersome to open in edit mode. However, I wonder if this is a browser limitation as when opened on a mobile device it's quite quick.

    Does anyone have any experience/tips/thoughts on report links?

    ------------------------------
    Mike Tamoush
    ------------------------------



  • 14.  RE: Apps, Performance, and Best Practice

    Posted 20 days ago
    Best practice 99% of the time is to set the form the way it defaults which is to have report links fields load the child tables separately and use Tabs for Desktop forms to allow Quick Base to present the form to the user with the minimum number of report link child tables populated.  That takes a lot of load to say run 8 reports and maybe the user only cares about  or two of those child tables at that moment.  They would probably load fast in mobile because they only have to pull the data when you click to view the report.

    If you can put embedded reports in Tabs, then the system will not try to pull the report(s) unless the user clicks the Tab. 

    Quick Base is single threaded for all basic operations, so if the form takes 3 seconds to populate then every other user is frozen until that form loads for those 3 seconds.

    ------------------------------
    Mark Shnier (YQC)
    Quick Base Solution Provider
    Your Quick Base Coach
    http://QuickBaseCoach.com
    mark.shnier@gmail.com
    ------------------------------



  • 15.  RE: Apps, Performance, and Best Practice

    Posted 20 days ago
    Mike -

    By workflows, I'm referring more to individual business processes / departments. As an example, I have a client that buys mineral rights. They originally had a single application that managed all functions within the company. After working like this for a year or two they found that the single app approach wasn't scalable and adding new functions often caused problems in unpredictable ways. That app still exists in a legacy capacity, but we have moved many functions into individual applications dedicated to: Sales, Personnel, Commissions, Corporate Planning, etc. They even have a completely separate application they use for planning and managing their Quick Base development. Many of their applications have some commonality, such as Employees and Customers and this information is pulled in via Sync tables. 

    On another note, your comment about running out of Form Rules on one of your forms tells me that you probably should consider breaking up that main form into multiple forms. Each user Role can have a different form for View, Edit, and Add and I would encourage you to take advantage of this feature.  This will allow you to have fewer Form Rules on each form and ensure that they always work properly.  Splitting these up, while it sounds like you'll just have more forms to manage, can actually make it easier and give you the opportunity to provide a better user experience tailored to a specific role.

    ------------------------------
    Blake Harrison
    bharrison@datablender.io
    DataBlender - Quick Base Solution Provider
    Atlanta GA
    404.800.1702 / http://datablender.io/
    ------------------------------



  • 16.  RE: Apps, Performance, and Best Practice

    Posted 19 days ago
    No it is just the size of the application and data within it without taking file attachments into account, I have a ton of file attachments myself. I think my biggest table is near 300 MB. That also goes into another issue though, the max table size of 500 MB. At some point I will need to figure out an archiving strategy before things hit that wall.

    I have one single table that has 1200 fields on it. It is awful to work in. When you load the field list page it takes a good 10-15 seconds to fully load and be useable.

    What Mark mentioned about multiple roles is a great idea and something we have started to implement as well. Rather than making one off roles for specific users we can have them in multiple so that they can have manager permissions or whatever it is while also performing their other duties. Starting this early is probably the easiest because once you get too many roles you get really deep into that swamp and it is hard to leave. I inherited my app at this company but it currently has over 100 roles in it, not a fun thing to work with. At this point trying to unravel that mess would be a huge undertaking.


  • 17.  RE: Apps, Performance, and Best Practice

    Posted 19 days ago
    Hi Mike,

    I just wanted to chime in and share a few resources we have that touch base on Best Practices around sharing data across apps. It goes into the various options and what their pros and cons are, minus Pipelines which came out after this guide.

    Sharing Data Across Quick Base Apps: Part 1 Cross App Relationships, Table to Table Imports, and Sync
    Sharing Data Across Quick Base Apps: Part 2 Quick Base Actions, Webhooks, Automations, and 3rd Party Integrations.
    Quick Base Memory Calculations
    How we built Quick Base to scale and you can too!

    As the other commenters have mentioned as an app gains in complexity and usage there is an increased chance of it needing more power and therefore having performance issues. Using methods like Sync, Automations, or Pipelines can help you avoid having apps directly connected but still be able to pass data back and forth and where possible it is a good best practice. In cases where I want to connect apps using these methods and still want to make it easy for users to move back and forth I have added buttons to the app dashboards that direct them to the right app/table to submit a form and buttons on the form to help them move back if needed. This way you can have a unified feeling experience but have ways to move data and users around.  I hope those resources are helpful. 




    ------------------------------
    Evan Martinez
    Community Marketing Manager
    Quick Base
    ------------------------------



  • 18.  RE: Apps, Performance, and Best Practice

    Posted 19 days ago
    Edited by dmlaycock2000 dmlaycock2000 19 days ago
    Hi Evan 

    These resources are excellent - thanks for sharing. 

    Could I ask you to clarify a point you make in the first document referencing the table to table imports, specifically:

    "TTIs create dependencies between applications when executed. Executing TTIs result in the share of system resources. "

    Is this basically saying that the depency and sharing of resources only occurs for the duration of the import (ie when it is actually executing the import)?

    So in my use case, I refresh my slave tables over night when there is almost no usage on either my master or slave app, so the shared resources and any performance issue on either app would not be an issue.

    Sorry if it seems like I'm basically asking the same question repeatedly, but I don't want to go rebuilding the way our apps work if I've simply misunderstood the nature of the dependency.

    Thanks 

    David

    ------------------------------
    dmlaycock2000 dmlaycock2000
    ------------------------------



  • 19.  RE: Apps, Performance, and Best Practice

    Posted 19 days ago
    Hi David,

    Not a problem at all it isn't always clear with TTIs how it all works, that is correct when a table to table import takes places the two apps that are involved get pulled together into one server in order to exchange that data. They will then stay there until the next time they would move around in the servers which isn't always on a fixed interval. If there is a saved table to table import they aren't able to separate from each other as the system respects that connection. One way around this that I have used which is a bit of a work around is to have a Scheduled Sync that syncs the data into a hidden sync table in the second app, then I use Automations or a Pipeline set to go off whenever records in the hidden Sync are added or modified and have it send all that new data over to my normal manual entry tables. This way I get the combo of a Sync table which doesn't cause the apps to be connected and the added bonus of all the data being put into my usual workflow where users can modify and act on it as needed.

    ------------------------------
    Evan Martinez
    Community Marketing Manager
    Quick Base
    ------------------------------



  • 20.  RE: Apps, Performance, and Best Practice

    Posted 19 days ago

    Yet again not the answer I was hoping for! But really appreciate the clarity thanks, Evan.  


    David



    ------------------------------
    dmlaycock2000 dmlaycock2000
    ------------------------------



  • 21.  RE: Apps, Performance, and Best Practice

    Posted 17 days ago
    Thanks to both Evan and Mark for your various responses on this thread.

    Given what I've now learned, it's clear that I need to transfer the data from my master to my slave app differently, so the two apps are run in different threads, for optimum performance.

    I thought Id share my plan here, in case there are any flaws which Mark, Evan and the community can point out for me, before I replace one problem with another!

    To summarise my current situation, I have 4 master tables in the Master App, and 4 slave tables in the slave app.

    The master app has lots of complex relationships (normal and reverse), summaries and lookups, and it's where we do our content management, so it's optimised for usability within the native QB UI. Lightning fast speed isn't a priority.

    The slave app is currently refreshed from the master on a daily schedule  (or upon manual trigger) by import API. 

    The slave is optimised for speed, has no relationships or formula fields, and only essential data for its sole purpose which is serving read only content to our web app.

    Given that I don't want to mess with the slave tables, and they cannot be reconfigured as connected tables to use the sync functionality, my plan is now to use a two step process as follows:

    1) Create 4 new connected tables in the slave app, to act as an interim store of the data. These interim holding tables would be refreshed using the connected tables sync functionality.

    2) Refresh my original real slave tables from the interim tables (both in the same app) using the good old import API.

    One thing at the back of my mind, that I'm not sure how I will approach yet (beyond careful scheduling) is...

    I'd like to trigger the API Import automatically when the sync has fully completed.

    And obviously this could involve anything from 1 to thousands of records changing, so I need to trigger this in a way which cannot result in a single import triggering the API calls multiple times.

    I also wouldn't want the import to happen until the sync had finished.

    Presumably I'll need to resort to pipelines for this, and I'm thinking that there will be no way to know the sync has finished, so it will have to just be a case of creating a pause in the processing that is long enough to be sure there is no way it wouldn't have.

    One final question for anyone who is familiar with pipelines. Presumably they run independently of QB db operations (except when they are running QB DB operations themselves). So a pause in a pipeline that is being run doesn't create a blockage in the processing of normal queries against the db? Is my assumption here correct?

    David

    ------------------------------
    dmlaycock2000 dmlaycock2000
    ------------------------------



  • 22.  RE: Apps, Performance, and Best Practice

    Posted 17 days ago
    Good plan.  A couple of thoughts.

    The refresh cycle is hourly and it will be at the same minute each hour.  You don't get to choose the minute but the History shows the time of completion.

    I don't know if it is possible that source table records will be deleted but if that is possible then you will need to consider how you will purge out those records are after you do the save table to table copy.  The Saved toTable  copy will copy across all records but not delete old ones.  

    Pipelines can be scheduled to run hourly and you do get to choose your minute, so a pipeline can run say every hours at 17 minutes after the hour if you are seeing that the Refresh completes at 15 minutes after the hour.

    You are correct that a dormant pipeline has absolutely no effect on the performance of an app. There is only load on the app when the pipeline is actually interacting with your app.

    ------------------------------
    Mark Shnier (YQC)
    Quick Base Solution Provider
    Your Quick Base Coach
    http://QuickBaseCoach.com
    mark.shnier@gmail.com
    ------------------------------



  • 23.  RE: Apps, Performance, and Best Practice

    Posted 17 days ago

    Brill, thanks Mark.

    D



    ------------------------------
    dmlaycock2000 dmlaycock2000
    ------------------------------