Let users take the wheel: How customers course-corrected our latest feature
A case study on preemptively solving customer problems through their behaviour
A feature that recently made its way onto our product roadmap at Visibuild was the capability to export multiple PDFs for Visis through any project location page, aptly named Bulk PDF Exports. Given our previous updates that enabled users the ability to filter Visis based on a chosen location, the next logical step was to empower users to export that data.
Note: A Visi is the broad term we use to describe Tasks, Inspections or Issues within the product”
Working at a startup poses interesting challenges for those working in product development, and kicking off a new project requires our team to come together and realign on the feature and its place on our priority list, particularly given the small size of our product team!
Entering the market quickly and bringing your product up to competitive feature parity and building out the “why we’re different” moat requires ruthless prioritisation as well as some creative, short feature-release timelines with a willingness to roll-forward or pivot at short notice.
Entering the market quickly and bringing your product up to competitive feature parity and building out the “why we’re different” moat requires ruthless prioritisation.
For each product feature we kick-off, we as a team make some best-effort estimations around a few key trade-offs:
- Time-to-market.
- Feature validation.
- Feedback cycles from the end-user.
There is always a fine balance between feature release and engineering. Underbake the feature and you risk breaching trust between your customers and your new feature. Spend too long in the oven and your customers may already have found alternatives that satisfied their immediate needs.
At Visibuild, we are working closely with our customers and remain transparent so that our finger is on the pulse and we can prioritise what needs to be done as effectively as possible. Once these decisions have been made, the internal team works together to put together the vision of the feature and work backwards on slicing up the end-goal into actionable pieces that help to bring value to the customer as soon as possible. Part of that process includes our efforts to predict things that could go wrong, how we can preemptively track those outcomes and also how we can roll forward in the case that the issue is considered a “deal breaker” on the promise that the feature makes.
One feature that I recently built out was the ability for customers to bulk export PDFs of our customers “Visis” (a universal umbrella term that covers the project’s inspections, issues, tasks and non-conformance reports).
This feature was heavily requested, and we wanted to slice up the end-result into iterations that enabled us to get the feature out to users quicker.
We sliced up the iterations for this feature into two pieces:
- The first iteration would introduce the customer-facing UI on the web application and utilise the backend flow we already had for emailing out PDF exports for a single Visi as an email attachment.
- The second iteration would focus on replacing the emailed zip files with the PDFs, storing those zip files remotely and replacing the email attachment with a link to the downloads.
The problem that we identified with the first flow is the limitation of email attachments. There is an attachment size limit for the emails that we know we would eventually hit as project data becomes larger in size and generating exports for those Visis becomes more complex in both size and nature.
As users begin to use our feature more, the likelihood of running into this issue increased significantly. Some of the factors that went into this (that we could identify) would include the number of attachments for a Visi, as well as the amount of Visis requested for the export.
Given this known assumption when defining our first iteration, we capped the bulk exports to have a maximum of fifty Visis that could be requested for export at one time. This cap was not designed to put a cap on our customers, but we knew that measuring this and enforcing the limit would give us the opportunity to release the feature earlier and collect usage statistics to help us make an informed decision on the second iteration to move to the download links. The cap would not prevent the possibility of a failure due to the attachment size being large, but given the arbitrary nature of the PDF attachment size, we figured that it would certainly help mitigate the risk of too many failing exports.
The decision of the cap afforded our product team some breathing space which in turn gave us the chance to spike out the solution for the second iteration of replacing the zipped PDFs while getting value out to customers faster.
Statistics from the early adoption
At Visibuild, our aim is to let customer feedback and tracked actions drive our product direction.
Given the agreed upon limitations of the email attachment for the first iteration, tracking was added in to view which projects were using the new feature, how many Visis they were attempting to export as part of the request and a way to capture the errors if exports became too large for email attachments.
As it turns out, even within the first week, bulk exports were used to export in larger batches than initially hypothesised.
The above graph displays the number of Visis requested for PDF generation per request. Within the first week, there were some early signs that users were requiring the full extent of the new feature’s capped export limit.
Within the first week, there were some early signs that users were requiring the full extent of the new feature’s capped export limit.
One request even managed to run into the exact scenario that we wanted to avoid: failure to send an email because of the ZIP file attachment hitting the size limit.
Our support team worked on ensuring that the failed export was supplied to the user who made the request, but we also identified and understood prior to release that issues such as this would be costly for our support team if they continued to occur, particularly as feature adoption would only increase exponentially.
Ultimately, we only had one failure from the first one hundred requests. Our decision to release the first iteration still worked in our favour. We had delivered value to customers for 99% of requests. That being said, this early usage, direct customer feedback and the first incident were all references that informed our decision to bring forward the next iteration of using download links over attachments up the priority list.
Early usage, direct customer feedback and the first incident were all references that informed our decision to bring forward the next iteration.
Rolling forward into the second iteration
At the end of first iteration, the workflow for exporting multiple Visis could be simplified into the following state chart:
The agreed upon technical solution to take this forward to the next iteration meant that the following changes had to be made:
- Introduce a way to track the state of a bulk export job i.e. is the job pending, in progress, fulfilled or rejected?
- Ensure notifications occur for both successful and unsuccessful jobs to keep the end-user informed.
- Build a secure mechanism to store and access exported ZIP files.
The next iteration would update our workflow could then be simplified to the following state chart:
This solution meant that we would resolve the issue around attachment sizes, as well as be more proactive on letting customers know if there was an issue that occurred during the bulk export process.
The final solution
After working with the rest of the product team to define the next iteration of the user experience, we implemented an updated set of email flows to keep the user informed. We also defined, implemented and tracked a set of states that the job could be in and how we could best reflect that through the user interface on our export download page to ensure that any further issues would be front-of-mind and addressed by our product team.
The new flow would start with the user requesting a new export and the success notification positioning the time frame for the export to give the user a better idea of how long exports may take.
Once the export had completed successfully and was ready to download, an updated email would now provide the link as opposed to an attachment. This addresses our issue on the attachment sizing problem that we ran into. The link would direct a user to our new exports download page which would give them information on the current state of the export and a link to download the export in the event that it was ready (and not expired).
Clicking the download link would provide the download page for the user to download the PDF.
Alongside the download link, we include the time until the link would expire. The decision was made to ensure that we set the link to “expired” after 14 days given that exported PDFs generally move into a stale state soon after the export.
This new approach with the expiry built-in enabled us to think forward on the costs of hosting these PDF export requests as remote ZIP files. This helped us to budget the new feature at scale, with our backend set to automatically remove these stale ZIP files and save on costs for data that would no longer be used.
This new approach with the expiry built-in enabled us to think forward on the costs of hosting these PDF export requests as remote ZIP files. This helped us to budget the new feature at scale.
In the case of a failure, we also had an email set to inform the user that an issue occurred to resolve the issue of “silent” failures causing customers to contact us about issues. This enabled us to be proactively transparent with a customer, as well as giving them the option to contact us for further information if desired.
The final piece of the puzzle was to keep the user interface for our export download page informative with the different job states that it could be in. For us, that meant ensuring the user was informed if a job was queued, in progress, failed, expired or in an unexpected error state.
Some of those states were represented in the user interface as the following:
Thanks to the updated user interface, we could ensure that users were empowered to check the progress of any given export request and understand where that request sits at any stage of the job lifecycle.
The updated lifecycle of the our new flow over time could be simplified and summarised as the following:
This flow gives an overview of the lifecycle for all our PDF exports as they occur for single PDF exports, bulk PDF exports (more than one PDF export in a request) and the failure flow for both the single and bulk PDF exports as those flows run over time.
Recount, outcome and results
At the time of writing, our iteration on the feature has been shipped and we have kept a close watch on the outcome and results.
To recount, we began with the first iteration that focused on getting the capability out to customers while keeping the end in mind in order to understand the next steps while keeping the roadmap fluid enough to action those steps at the appropriate time. After considering usage analytics and customer feedback, we promoted the implementation of our second iteration up the roadmap.
This second iteration had an end-goal to solve and workaround the issue of attachment sizes and work towards unlocking the hard Visi export limit that we have set for customers. We did this by migrating our previous approach of directly attaching exported ZIP files to emails as attachments and instead opted to make use of an interim storage solution and download links.
Thanks to this latest iteration, we have successfully mitigated these issues and no more requests have come in around failed exports due to the attachment size.
In the time since the release of this new iteration, adoption of the feature has increased over 370%. Our decision to move up the iteration on the roadmap has likely prevented a few headaches and tough conversations with our customers.
This project showed the collaborative mentality we share at the company. By keeping the end in mind, we preemptively acknowledge potential pitfalls in early iterations, as well as re-enforce our company value to pivot and address any high-risk and problematic issues if they occur earlier than anticipated.
By keeping the end in mind, we preemptively acknowledge potential pitfalls in early iterations, as well as re-enforce our company value to pivot and address any high-risk and problematic issues if they occur earlier than anticipated.
The two iterations of the bulk PDF export feature that were covered in this recount demonstrate the importance of shipping value early and making decisions based on customer usage.
Running into the issue of users wanting to make the most of a feature is a good problem to have. As is often said in our team offices, “Ship to learn.”
