Telemetry and insights into your product

Something that’s quite incredible in this modern, ever-connected world, if you’re working on a product team, is that you could be getting a wealth of insights from the thing you’re building. It must have been hard in the days where you had your product run on isolated machines and never heard back from it after you shipped – except for when there’s a problem and support tell you that things are broken. You may have had a few close customers or users, your forums or other contact channels, but you were dependent on data from others and you’d get only the things they shared with you.

Things got a little better when you were able to assume most users and machines were connected to the internet and that you could plan for some signals and data to come back to you. If you chose an architecture that required access to cloud services, that made it easier to hear back from your product – and you at least had an idea about which versions were still “in the wild”, what you needed to continue to support, how fast your users and customers updated from version to version.

Cloud services that offer the vast majority of capabilities, if not all of them, “as a service”, runs completely in your control: all microservices, code, API and user experiences are hosted by you and through you, so you’re in full control over the code and what you collect, from a usage perspective – that then helps you, to make smarter decisions. Clearly, what you collect has boundaries – and you’re bound to GDPR and other regulation to make sure you adhere to privacy rules and laws. But assuming you are keeping your hands off PII data and any data that is tied to single (named) user’s behavior, or even single customer organization behavior, and it’s all anonymous, you could learn a whole lot more about your product:

• What needs prioritization: based on usage numbers (how often was X used, compared to Y, how many users in total, how much volume, etc.). You may want to prioritize what customers love and what they buy your product for. Knowing what scenarios and features your users go through most – or better yet: knowing which customer audiences and segments prioritize which features from you, helps fine-tune features, UX, documentation and marketing.
• What could be or should be moneitized: once you know what customers use, you may also learn what volume they generate, and if there’s customer audiences that use your service or feature significantly more or busier than others. This may indicate that there could be a way of monetization, by either allowing larger volumes or faster processing for paying customers, or simply additional features for specific customer segments that may be expensive to build or operate – because they’d require additional infrastructure or extra compute.
• Find out why a feature isn’t used and where “drop offs” happen: if you’re lacking facts as to why your feature isn’t being used, adding telemetry and pointers that indicate where users drop off and what areas of the product they don’t use, may give you clues. Potentially click rates are low because the feature can’t be found, or the assistant that walks the user through an experience is confusing.
• Enhancements – A/B testing: During a new release or possibly also pre-release, you could collect extra data and compare the before with the after – and gauge value early on. You would also be able to figure out if there need to be adjustments to make the feature or service successful on release. You could run A/B testing and collect data, and compare the results from the two experiences.
• Games: balancing the progression: I noticed achievements on game platforms as a neat way of following along with how players progress with your game after launch. Since more and more games are now released early on and finished with the players and community, Collecting the right telemetry that gives you performance hints, records and collects user journeys and helps you fine-balance different parts of the game, early-game, mid-game, end-game fairly well.

I am excluding “Health” as a point here on purpose – that’s usually a discipline that’s fairly well under control by your colleagues from Engineering, if they build, run and operate the services, features and infrastructure that make up your product. There should be data and telemetry that ensures that you learn when roundtrip times go up, request numbers go significantly down — something needs to be fixed.

Clearly the possibilities and telemetry you want to gather highly depends on the kind of software or service you are building and what your plans are. Without doubt, there’s great power in being able not only to understand what telemetry you currently are collecting and making sense of it, but also to define what data and metrics you need to collect. When planning ahead, your specification and requirements gathering should also include a deliberate idea of what data points you need to make smart decisions moving forward with the feature – and validate or falsify ideas that you had. Next to the actual feature, behind the scenes, the right data should be collected as users, customers, partners use your service, that allow you to answer the next questions you have for the next step.