Virtually two years in the past, we laid out our concept of recent metadata for the trendy information stack, the Information Catalog 3.0.

Since then, it’s been a dream seeing our imaginative and prescient come to life. Lively metadata has change into a class and central speaking level within the information world. Gartner, Forrester, and G2 alike made main strikes to scrap outdated concepts of knowledge cataloging and introduce new ones round energetic metadata and fashionable cataloging.

This transformation was pushed by actual product developments, ones that turned this summary dream into one thing that information groups might really use.

In 2022, our engineering crew shipped over 200 new options, together with a model new model of our product. These enhancements proved the facility of energetic metadata and even acquired us ranked as a Chief within the Forrester Wave, Q2 2022!

Earlier than we tackle the subsequent 12 months of recent options, let’s take a couple of minutes to look again on an incredible 12 months. Listed here are 15 of our favourite options from 2022.

TL;DR: 15 updates in Atlan you need to find out about

From siloed instruments to embedded context

We now have at all times been pushed by one core concept — range is the one actuality. Information groups are one of the vital numerous groups immediately, from the instruments they use and the folks they work with to their very own use instances and personas. This range is each a energy and a weak point.

Information groups work throughout a various set of instruments — BI instruments like Looker, collaboration instruments like GitHub, communication instruments like Slack, ticketing programs like Jira, and extra. Too usually, these flip into siloed information programs. Should you’re in a dashboard, you need to know when you can belief it then and there, quite than switching and looking out throughout different instruments.

In 2022, we constructed range into the core of our product. By activating metadata, Atlan sends essential context again into the workflows and instruments the place the people of knowledge work every single day.

Improve collaboration and seize information with our Slack integration

With our Slack integration, launched in January 2022, we set the stage for in depth collaboration on the earth of metadata. Atlan’s deep integration with Slack goes past simply sending messages — it’s about sharing and capturing necessary information throughout your group.

Atlan’s Slack integration contains:

• Wealthy preview hyperlinks: Share property along with your crew on Slack and see detailed previews.
• Workflow alerts: Get alerts in Slack every time an Atlan workflow fails or succeeds.
• Ask a query from Atlan: Ask a query by clicking on the Slack icon for any asset. That Slack dialog will even be documented mechanically in Atlan for future reference.
• Doc threads instantly from Slack to Atlan: Simply share Slack conversations to particular property in Atlan.
• Search phrases and queries instantly from Slack: Seek for phrases and queries in Atlan with out leaving Slack.

Keep in circulate with our Atlan Chrome Extension

Lively metadata makes it potential for metadata to circulate effortlessly and rapidly throughout your complete information stack, embedding enriched context and data in each instrument within the information stack.

Think about you’re taking a look at a dashboard and need to know who owns it and what a metric means. How would you go about it? Nicely, you’d most likely have to determine who the proprietor was. Then you definately would most probably take a display seize of the dashboard, electronic mail or Slack it to that proprietor, and ask your questions. Not anymore!

Now you possibly can skip all these steps by merely clicking on the Atlan icon in your browser. It would open the Atlan Chrome extension proper in your dashboard, supplying you with all of the context you want proper if you want it. Atlan makes this potential within the instruments the place your information groups and information customers spend most of their time — Looker, Tableau On-line, Snowsight, Mode, Energy BI, BigQuery, and Salesforce, with extra instruments on the best way.

Clear up points quicker with our Jira integration

Monitoring points and fixing issues simply acquired quite a bit simpler with our Jira integration. For a lot of information groups, attending to the “root trigger” of damaged dashboards or inconsistent numbers is usually a lengthy, drawn-out course of. Not anymore!

With Atlan’s Jira integration, customers can now rapidly open a brand new Jira ticket or handle an current ticket instantly from any Atlan asset. Simply click on the Atlan sidebar.

Reduce threat throughout information creation with our GitHub integration

When a knowledge engineer makes modifications to a dbt mannequin, they may unknowingly be affecting tens or a whole bunch of individuals, tables, or dashboards. How can we reduce this threat whereas additionally offering transparency for the information engineer and information client?

With our integration between Atlan and GitHub, information engineers can now entry all of the context they should reduce the chance for information customers.

Right here’s an instance of what this seems to be like: say that you just’re a knowledge engineer, and also you’ve created a pull request. When the GitHub motion runs, it is going to mechanically create a listing of all downstream property that will probably be affected by this request — earlier than you make the change. From there, you possibly can attain out to customers upfront, or analysis and take a look at the property to see how they may be affected.

From dispersed, handbook instruments to automated 360° context

Simply as information groups immediately use numerous instruments, their information ecosystem is made up of numerous information property. These are not simply tables made up of rows and columns. Information property are exams, metrics, pipelines, code, queries, dashboards, and extra — all scattered throughout totally different information instruments and programs.

At this time, if you onboard an engineer, you simply share a hyperlink to a GitHub repository. This has the code, documentation, historical past, collaborators, and every thing else you might want to know in regards to the code, multi functional place.

Nevertheless, it takes numerous time and handbook effort to arrange and convey documentation into conventional information catalogs immediately — and simply as quickly as you’re achieved, they’re old-fashioned. At Atlan, we needed to make information property simply as straightforward to share and perceive as software program on GitHub. Together with activating metadata again into day by day workflows, our workspace offers customers a 360-degree view of each asset by mechanically pulling in and stitching collectively metadata from each instrument in your information stack.

Atlan Playbooks: the Zapier for metadata

One of many frequent questions we get from information groups is, “How can we automate our metadata?” Whereas different groups like advertising or gross sales can simply do action-based automation at scale with instruments like Zapier or Salesforce, information groups usually don’t have the bandwidth to code customized automations for every numerous use case. Why can’t there be a Zapier for metadata?

That’s the reason we’ve got developed the primary low-code/no-code metadata automation for information groups. With Atlan’s Playbooks, customers can now create rule-based automations at scale. These can be utilized to drive infinite use instances throughout your group. Listed here are a couple of examples:

• Deprecate property: Mark any property that haven’t been queried within the final 30 days as deprecated.
• Add/change possession: For Salesforce property which can be lacking an proprietor, add RevOps because the proprietor.
• Report failed property: Publish a listing on Slack of each desk and dashboard with a “failed” Airflow standing.
• Shield delicate information: Connect GDPR customized metadata on all property tagged as “PII”.
• Flag upstream alerts: Notify downstream house owners when an upstream asset is tagged with a warning or announcement.

Cut back handbook documentation with Trident ideas

Conventional metadata administration is usually a tiresome, handbook course of that requires numerous human intervention from information leaders and stewards.

That’s why we launched Trident, which offers automated ideas for brand spanking new descriptions, house owners, phrases, and classifications. Trident makes metadata enrichment simpler in order that information groups can spend extra time on significant initiatives and fewer time writing descriptions and classifying information. Since its launch, one-third of all description updates on Atlan have been made with Trident!

P.S. We just lately gave a preview of the potential of Trident by combining it with GPT-3 to create Trident AI — more and more clever automated descriptions, enterprise phrases, READMEs, and extra.

Handle your information and minimize prices with asset recognition and utilization metrics

One of many largest challenges going through information groups immediately is managing their information and decreasing information infrastructure prices. That’s why we launched utilization and recognition metrics.

Information groups can now analyze their Snowflake information property with out leaving Atlan. With these new metrics, organizations can establish their most used property, discover folks with essentially the most context on property, establish unused property, and extra. These metrics embody:

• Asset question logs: The variety of occasions a knowledge asset has been queried (i.e. asset recognition), when it was final up to date, and so on.
• Person question logs: A knowledge asset’s prime customers and most up-to-date customers, who final queried it, and so on.
• Price and efficiency optimization logs: Which queries are slowest, costliest, and so on.

Get full visibility and column-level lineage with our Fivetran Metadata API integration

Atlan’s deep partnership and integration with Fivetran creates the primary end-to-end view of the trendy information stack. By leveraging Fivetran’s new Metadata API, Atlan now builds column-level lineage out of your Fivetran sources all the best way to BI instruments like Looker and Tableau — supplying you with full visibility throughout your fashionable information stack with highly effective root trigger and influence evaluation.

We’re excited to companion with Atlan to carry higher metadata administration to our prospects. Information governance is a big wrestle for information groups immediately.

Collectively, Fivetran and Atlan are fixing this drawback by serving to information groups achieve much-needed visibility on the place information got here from, who accessed it, and what modifications have occurred within the pipeline.

Fraser Harris, VP of Product at Fivetran

Make metrics and fashions first-class residents with our dbt Semantic Layer integration

One other huge partnership from this 12 months was Atlan + dbt. With our new Semantic Layer integration, dbt metrics and fashions at the moment are first-class residents in Atlan.

Atlan’s deep integration with the Semantic Layer brings dbt’s wealthy metrics and fashions into the remainder of the information stack, permitting information groups to create repeatable, metadata properties — like desk house owners and verified tags — and construct documentation requirements.

dbt’s metrics are additionally now part of Atlan’s column-level lineage, spanning from supply programs and information storage to transformation and BI. With elevated context, visibility, and self-service, this integration helps numerous information groups work collectively higher.

Get a birds-eye view of your information property with our Reporting Middle

At Atlan, we’ve seen a sample of questions come up as our prospects collect extra metadata. Our customers need to perceive how their information panorama and its metadata are evolving.

That’s why we launched the Reporting Middle, which offers a fowl’s-eye view of all of the metadata updates in Atlan via visualizations that will help you observe adoption, curation, and metadata administration. In brief, Reporting Middle offers you information in your metadata. It doesn’t get extra meta than this.

With the Reporting Middle customers can now:

• Monitor their property
• Observe glossary phrases and classes
• Implement governance by monitoring personas, functions, classifications and requests
• Perceive queries and question runs from insights in Atlan.

From generic instruments to personalised experiences

One other problem with conventional information catalogs and metadata options is that they deal with each person the identical, delivering a generic expertise throughout the group.

The issue with that strategy is, context means various things to totally different folks. For instance, a knowledge engineer seems to be at a desk and so they’re fascinated with transformations, information high quality, and pipeline stability. In the meantime, an analyst might care about column names and descriptions, frequency distributions, and in style queries ran towards that desk. Lastly, a enterprise person may simply see a metric and marvel “What does this metric imply?” and “Can I belief it?”

We needed to vary that narrative and construct a platform that delivers a customized expertise for every person throughout your group.

Create a Netflix-like information expertise with Personas and Functions

If Netflix can create a customized viewing expertise for his or her customers, we thought, why can’t Atlan do the identical factor for information groups? That’s precisely what we did with the introduction of personas and functions.

Personas and functions give admins the flexibility to regulate entry to property, insurance policies, and preferences tailor-made to the function of every particular person. Personas present asset-level controls for every crew, whereas functions group property collectively for cross-team initiatives.

For instance, the advertising persona might give all advertising folks entry to sure Snowflake tables and Tableau property. With a goal, some members of the advertising crew might also get entry to different advertising marketing campaign information and dashboards which can be related to campaigns they’re operating this 12 months.

We’ve seen this having every kind of attention-grabbing implementations inside organizations. For example, many information groups utilizing Atlan have began fascinated by implementing paradigms like the information mesh, which at its core has this idea of a website. Utilizing personas and functions, you possibly can create extraordinarily personalised experiences by area, which then permits your information mesh to start out being profitable.

Curate person experiences with customized metadata

Customized metadata is likely one of the most useful options in Atlan. Together with creating their very own metadata (like a knowledge freshness rating or information high quality metrics), admins can choose which customized metadata must be seen to every persona whereas limiting visibility to those who wouldn’t discover this info related.

For instance, it’s seemingly {that a} finance persona received’t want entry to dbt transformations and Airflow DAGs, however they could have to dig into lineage to troubleshoot an odd Salesforce report or question a Snowflake desk to find out ARR.

With customized metadata, Atlan offers the flexibility to curate every individual’s distinctive expertise, serving to them work extra effectively with the precise info and making certain they don’t get overwhelmed by your complete information property.

Energy personalised energetic metadata with an all-new Atlan person interface

At Atlan, we’ve at all times been dedicated to serving to information groups work collectively higher. We began out as a knowledge crew ourselves and failed 3 times in making an attempt to implement a knowledge catalog. So we determined to construct Atlan to be the type of platform and companion that we wished we had again then.

After over a 12 months of labor, we launched a model new model of Atlan final 12 months. From an entire redesign of the Atlan person interface to highly effective energetic metadata and personalization options, this replace made it simpler than ever for information groups to construct belief and democratize information with metadata.

From closed, siloed instruments to open programs

When you may have a various set of customers, utilizing a various set of instruments and property, all throughout industries, information stacks, and extra, there will probably be a range in use instances. Each firm’s use case is totally different, and one measurement doesn’t match all. That’s why we care about constructing open and extensible platforms that enable information groups to resolve their very own issues in their very own methods.

Drive infinite integrations and use instances with a totally open API structure

Offering a completely open API structure permits each motion to be programmatically pushed in Atlan. We now open all our metadata change occasions via AWS Eventbridge and a Kafka queue, Lamda webhooks, and extra.

On the core of the Atlan platform, every thing is open by default — and this enables information groups throughout any group to construct for and remedy all kinds of use instances. Listed here are a pair that we’ve got seen organizations begin to implement.

Programmatic information archival and deletion

One among our prospects, a media firm, purchases numerous exterior information. The info has an expiration date, in order that they arrange handbook processes to frequently handle and delete their expired information. This generally led to information not being deleted, leading to authorized and compliance dangers. This course of additionally pressured the information crew to frequently spend vital time on handbook work, leading to misplaced productiveness and crew output.

Now, utilizing the Atlan API, a notification is triggered 30 days earlier than the contractual finish date for the information. This creates a Jira ticket, letting the suitable events know that the information can then be deleted or archived. By this new programmatic metadata workflow, the corporate was not solely in a position to eradicate authorized threat but additionally save vital time and productiveness.

Lively information belief administration

One other buyer has built-in Atlan with a knowledge high quality instrument and energy energetic information belief administration. They use our open APIs to usher in anomalies which can be detected by the information high quality instrument, after which downstream BI property and probably impacted enterprise metrics are mechanically flagged. For instance, if a desk feeding a KPI has a knowledge high quality situation, Atlan mechanically creates an Announcement for the KPI within the Atlan enterprise glossary.

Construct customized event-based automations with our AWS EventBridge integration

Many groups throughout organizations, from information observability to incident administration, are constructing automations that may set off actions primarily based on occasions. However there’s by no means been something obtainable for metadata administration to create these kind of event-driven use instances for metadata… till now.

Information groups can now use Atlan to create production-grade, event-driven automations for the world of metadata. This leverages our integration between Atlan and AWS EventBridge, an AWS service that creates an occasion and lets customers devour and construct use instances on it, to create Atlan metadata occasions into an EventBridge account.

Listed here are some examples of how this may work in Atlan:

• Possession alerts: Get notifications in Slack when there’s a change within the possession of an asset.
• Propagation and classification: If somebody marks a discipline as PII in an upstream information supply, mechanically create a masking coverage to vary all associated fields downstream.
• And numerous extra use instances, reminiscent of notifications round schema modifications on property in Atlan for information engineering, login/logout occasions for safety groups, or triggering a Fivetran enrichment occasion to kick off an Atlan workflow.

—

💡 To be taught extra about the entire releases for 2022, try shipped.atlan.com. You can too subscribe to our Product Updates e-newsletter for all the most recent information.

💡 To take a look at Atlan for your self, go to our product tour.

💡 Prepared to start out utilizing these new options? Attain out to our Gross sales crew or your Buyer Success Supervisor to learn how.