JSON Handler - Max JSON size exceeded

Question

Hello all.
I wonder if anybody could offer any advice on a problem i'm having please?
I have set up an 'Fetch a JSON resource' in pipelines, which is to a third party ID check. Prior to the candidate completing the ID check I can fetch the JSON data and import it into QB through Pipelines. However, once the candidate completes the ID check the JSON contains considerably more fields and when I try to 'Fetch a JSON resource' I get a validation error.
Validation error: Maximum JSON size exceeded
Is there any way in which I can overcome this please? Here is an example of what is in the JSON.
[&nbsp; {&nbsp; &nbsp; "id": "e4x69a61-07mp-46le-aj80-0c52f12a673e",&nbsp; &nbsp; "data": {&nbsp; &nbsp; &nbsp; "pseudonym": "ANON-e2x0a6mp-37l0-42ps-94ef-u69755438d3o",&nbsp; &nbsp; &nbsp; "product_name": "Test Product",&nbsp; &nbsp; &nbsp; "status": "completed",&nbsp; &nbsp; &nbsp; "purchase_start_date": "2022-03-16T09:33:52Z",&nbsp; &nbsp; &nbsp; "purchase_completed_date": "2022-03-16T09:33:52Z",&nbsp; &nbsp; &nbsp; "date_of_birth": "1974-01-17",&nbsp; &nbsp; &nbsp; "first_name": "JOHN",&nbsp; &nbsp; &nbsp; "gender": "-",&nbsp; &nbsp; &nbsp; "surname": "DOE",&nbsp; &nbsp; &nbsp; "liveness_verified_selfie_1": [&nbsp; &nbsp; "iVBORw0KGgoAAAANSUhEUgAAAJYAA...MTkrMDA6MDDuShMbAAAAAElFTkSuQmCC
"],&nbsp; &nbsp; &nbsp; "liveness_verified_selfie_2": [&nbsp; &nbsp; &nbsp; "iVBORw0KGgoAAAANSUhEUgAAAJYAA...R24Ac1JHQrqQcwcAAAAASUVORK5CYII=
"],&nbsp; &nbsp; &nbsp; "driving_licence_hd_attachment": null,&nbsp; &nbsp; &nbsp; "id_card_hd_attachment": null,&nbsp; &nbsp; &nbsp; "passport_hd_attachment": [&nbsp; &nbsp; &nbsp; &nbsp; "iVBORw0KGgoAAAANSUhEUgAAAJYAA...MTkrMDA6MDDuShMbAAAAAElFTkSuQmCC
",&nbsp; &nbsp; &nbsp; &nbsp; "iVBORw0KGgoAAAANSUhEUgAAAJYAA...R24Ac1JHQrqQcwcAAAAASUVORK5CYII=
"],&nbsp; &nbsp; &nbsp; "age": 49,&nbsp; &nbsp; &nbsp; "age_is_18_plus": true,&nbsp; &nbsp; &nbsp; "driving_licence_attachment": null,&nbsp; &nbsp; &nbsp; "driving_licence_expiration_date": null,&nbsp; &nbsp; &nbsp; "driving_licence_number": null,&nbsp; &nbsp; &nbsp; "driving_licence_state": null,&nbsp; &nbsp; &nbsp; "driving_licence_verified_at": null,&nbsp; &nbsp; &nbsp; "full_name": "GARY GIRAFFE",&nbsp; &nbsp; &nbsp; "id_card_attachment": null,&nbsp; &nbsp; &nbsp; "id_card_expiration_date": null,&nbsp; &nbsp; &nbsp; "id_card_number": null,&nbsp; &nbsp; &nbsp; "id_card_state": null,&nbsp; &nbsp; &nbsp; "id_card_verified_at": null,&nbsp; &nbsp; &nbsp; "liveness_verified_selfie_1_at": "2023-04-11T11:17:03.386Z",&nbsp; &nbsp; &nbsp; "liveness_verified_selfie_2_at": "2023-04-11T11:17:03.386Z",&nbsp; &nbsp; &nbsp; "middle_name": null,&nbsp; &nbsp; &nbsp; "passport_attachment": [&nbsp; &nbsp; &nbsp; &nbsp; "iVBORw0KGgoAAAANSUhEUgAAAJYAA...MTkrMDA6MDDuShMbAAAAAElFTkSuQmCC
",&nbsp; &nbsp; &nbsp; &nbsp; "iVBORw0KGgoAAAANSUhEUgAAAJYAA...R24Ac1JHQrqQcwcAAAAASUVORK5CYII=
"],&nbsp; &nbsp; &nbsp; "passport_expiration_date": "2028-07-09",&nbsp; &nbsp; &nbsp; "passport_number": "1234567890",&nbsp; &nbsp; &nbsp; "passport_state": "accepted",&nbsp; &nbsp; &nbsp; "passport_verified_at": "2022-01-20T09:10:25.154Z"&nbsp; &nbsp; }&nbsp; }]
Thank you,
Michael
------------------------------Michael White------------------------------

chayceduncan · Answer

Does the API you're calling have additional parameters where you can limit/query only for certain fields? If so, I would try and minimizing the scope of your JSON response in the call itself only to the necessary fields and make sure that your query is only for a single object. Your only real avenue with how you're trying it is to minimize the response and get the size down.&nbsp;
The only other idea I can imagine is to forego using the JSON resource fetch and use the generic webhook to get the info, in my experience the parsing of that response is reasonably the same but I've never had it return payload size issues.&nbsp;------------------------------Chayce Duncan------------------------------

michaelwhite1 · Answer

Hi Chanyce.
Thank you for the reply.
I have been trying to limit the payload, but i'll be honest I've no idea how to do that.
You can see from below the size of each 'item'. I don't actually need the four large images which are causing the issue, but i've no idea how i can call the payload just for the other items.

Here is how i've set all the fields in the JSON handler 'fetch' request.

Have you got any ideas please?
Many thanks,
Michael------------------------------Michael White------------------------------

chayceduncan · Answer

Can you tell me what system you're trying to integrate with so I can take a look at the documentation and see if there is a way to limit the scope?
------------------------------Chayce Duncan------------------------------

michaelwhite1 · Answer

Hi Chayce.
The third party is Digidentity. The documentation can be found here: https://docs.digidentity.com/#aae3b6c2-c2b0-4250-9335-9a402eaf3dbd
It's under 'Onboarding', 'One-time verification', 'Retrieve report'.&nbsp;
Thank you,
Michael------------------------------Michael White------------------------------

chayceduncan · Answer

From reading through the docs on that call and looking at others, the only thing that might work is to alter the scope when you're getting the credentials. I can't find anything specific to that report request you're doing, but thats the only place where it would be possible to alter the response potentially but also changing it might interfere with the ability in general just to get the report you're querying.&nbsp;
'

The only other thought might be to try to use the Webhook Channel / Make Request step. The general setup will be identical to what you're currently doing with the JSON request, but it might not come with the same response payload max size. ------------------------------Chayce Duncan------------------------------

Forum Discussion

JSON Handler - Max JSON size exceeded

10 Replies

Related Content

Pipeline Make Request - Return Make Request JSON Output into fields

Pipelines JSON Handler - Fetch JSON URL

Pipeline JSON Issue

Pushing JSON data to external API endpoint

429 Error: Webhooks are exceeding the rate threshold

Recent Discussions

Empower Hour Tomorrow!

When the Expiration Date is On or Before Today, Change the Contract Status to Expired

Rollup Help

New Forms Return and Cancel Buttons Glitch

How to handle error in Upload step in Pipeline