Data Ingestion & Onboarding

Problem

We ran a research project throughout 2024 to understand the main pain points that Elastic Observability users experience during onboarding. Over thirty user interviews were conducted as part of this project. Key findings:

• Too many options: Users are unclear whether to use Beats or Elastic Agent (Fleet or standalone), with no clear in-product guidance on the most appropriate shipper for a given scenario
• Complex setup flows: The Fleet-based onboarding flow is difficult to follow for first-time users and requires significant technical knowledge of the Elastic stack
• Agentless onboarding gaps : Cloud provider integrations are not at par with other major vendors

Goal

Create a single landing page for data onboarding with clear categorization of use cases, easy access to relevant docs, and opinionated placement of recommended integrations. Users should go from curious to confident in under 60 seconds.

Solution

Designed a unified "Add Observability data" landing page with a guided decision assistant:

• Four clear categories: Host, Kubernetes, Application, and Cloud, with relevant technology logos.
• Searchable integrations browser to find specific data sources quickly.
• "Quick Start" options for common scenarios like Auto-detect Integrations with Elastic Agent.
• Opinionated recommendations reducing cognitive load of choosing between shippers.
• Promotion of OpenTelemetry where applicable.

Prioritized use cases based on telemetry: Kubernetes, AWS Firehose, APM, and Kubernetes with OpenTelemetry.

Outcome

• 60%+ tile click rate on the landing page, showing strong interest
• 30–45% completion rate for users who engaged with the setup flow
• Setup instructions are executable, users copy commands, data flows in successfully

Key insight: Users lack confidence, not intent. The biggest drop occurs before copying the install command.

The guided assistant addresses this by qualifying intent quickly, reducing choice to one clear path, and setting expectations before install.

Next steps: Future work will focus on solving concrete jobs-to-be-done and better understanding user environments, exploring how AI Agents can assist with onboarding, troubleshooting, and configuration tasks.